Augmented reality system and method for spectroscopic analysis

ABSTRACT

Wearable spectroscopy systems and methods for identifying one or more characteristics of a target object are described. Spectroscopy systems may include a light source configured to emit light in an irradiated field of view and an electromagnetic radiation detector configured to receive reflected light from a target object irradiated by the light source. One or more processors of the systems may identify a characteristic of the target object based on a determined level of light absorption by the target object. Some systems and methods may include one or more corrections for scattered and/or ambient light such as applying an ambient light correction, passing the reflected light through an anti-scatter grid, or using a time-dependent variation in the emitted light.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional Application Ser.No. 62/646,262, filed Mar. 21, 2018, entitled AUGMENTED REALITY SYSTEMAND METHOD FOR SPECTROSCOPIC ANALYSIS, the entirety of which isincorporated herein by reference.

INCORPORATION BY REFERENCE

This application incorporates by reference the entireties of each of thefollowing US patent applications: U.S. patent application Ser. No.15/072,341; U.S. patent application Ser. No. 14/690,401; U.S. patentapplication Ser. No. 14/555,858; U.S. application Ser. No. 14/555,585;U.S. patent application Ser. No. 13/663,466; U.S. patent applicationSer. No. 13/684,489; U.S. patent application Ser. No. 14/205,126; U.S.patent application Ser. No. 14/641,376; U.S. patent application Ser. No.14/212,961; U.S. Provisional Patent Application No. 62/298,993(corresponding to U.S. patent application Ser. No. 15/425,837); U.S.patent application Ser. No. 15/425,837; and U.S. Provisional PatentApplication No. 62/642,761.

BACKGROUND Field

The present disclosure relates to systems and methods for augmentedreality using wearable componentry, and more specifically toconfigurations of augmented reality systems for identifying material byreflective light properties.

Description of the Related Art

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; and anaugmented reality or “AR” scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user while still permitting the user tosubstantially perceive and view the real world.

For example, referring to FIG. 1, an augmented reality scene (4) isdepicted wherein a user of an AR technology sees a real-world park-likesetting (6) featuring people, trees, buildings in the background, and aconcrete platform (1120). In addition to these items, the user of the ARtechnology also perceives that he “sees” a robot statue (1110) standingupon the real-world platform (1120), and a cartoon-like avatar character(2) flying by which seems to be a personification of a bumble bee, eventhough these elements (2, 1110) do not exist in the real world. As itturns out, the human visual perception system is very complex, andproducing a VR or AR technology that facilitates a comfortable,natural-feeling, rich presentation of virtual image elements amongstother virtual or real-world imagery elements is challenging. Forinstance, head-worn AR displays (or helmet-mounted displays, or smartglasses) typically are at least loosely coupled to a user's head, andthus move when the user's head moves. If the user's head motions aredetected by the display system, the data being displayed can be updatedto take the change in head pose into account. Certain aspects ofsuitable AR systems are disclosed, for example, in U.S. patentapplication Ser. No. 14/205,126, entitled “System and method foraugmented and virtual reality,” which is incorporated by reference inits entirety herein, along with the following additional disclosures,which relate to augmented and virtual reality systems such as thosedeveloped by Magic Leap, Inc. of Fort Lauderdale, Fla.: U.S. patentapplication Ser. No. 14/641,376; U.S. patent application Ser. No.14/555,585; U.S. patent application Ser. No. 14/212,961; U.S. patentapplication Ser. No. 14/690,401; U.S. patent application Ser. No.13/663,466; U.S. patent application Ser. No. 13/684,489; and U.S. PatentApplication Ser. No. 62/298,993, each of which is incorporated byreference herein in its entirety.

Systems and methods disclosed herein address various challenges anddevelopments related to AR and VR technology.

SUMMARY

Various implementations of methods and apparatus within the scope of theappended claims each have several aspects, no single one of which issolely responsible for the desirable attributes described herein.Without limiting the scope of the appended claims, some prominentfeatures are described herein.

In some embodiments, a wearable spectroscopy system is provided. Thewearable spectroscopy system comprises a head-mounted display systemremovably mountable on a user's head, one or more light sources coupledto the head-mounted display system and configured to emit light in anirradiated field of view, one or more electromagnetic radiationdetectors coupled to the head-mounted display system and configured toreceive reflected light from a target object irradiated by the one ormore light sources within the irradiated field of view, one or moreprocessors, and one or more computer storage media. The one or morecomputer storage media store instructions that, when executed by the oneor more processors, cause the system to perform operations comprisingcausing the one or more light sources to emit light, causing the one ormore electromagnetic radiation detectors to detect light from theirradiated field of view including the target object, determining anambient light correction by detecting ambient light levels, applying theambient light correction to the detected light to determine levels oflight absorption related to the emitted light and reflected light fromthe target object, identifying, based on the levels of light absorption,a characteristic of the target object, and displaying the identifiedcharacteristic to the user on the head-mounted display system.

In some embodiments, a wearable spectroscopy system is provided. Thewearable spectroscopy system comprises a head-mounted display removablymountable on a user's head, one or more light sources coupled to thehead-mounted display system and configured to emit light in anirradiated field of view, one or more electromagnetic radiationdetectors coupled to the head-mounted display system and configured toreceive reflected light from a target object irradiated by the one ormore light sources within the irradiated field of view, an anti-scattergrid disposed between the electromagnetic radiation detector and thetarget object, the anti-scatter grid configured to attenuate at leastone of scattered light and ambient light incident thereon, one or moreprocessors, and one or more computer storage media. The one or morecomputer storage media store instructions that, when executed by the oneor more processors, cause the system to perform operations comprisingemitting, from the one or more light sources, light of a firstwavelength in an irradiated field of view, detecting, at the one or moreelectromagnetic radiation detectors, light of the first wavelengthreflected from a target object within the irradiated field of view,identifying, based on an absorption database of light absorptionproperties of at least one material, a material characteristic of thetarget object, and causing a graphics processor unit to display, to theuser, an output associated with the material characteristic.

In some embodiments, a wearable spectroscopy system is provided. Thewearable spectroscopy system comprises a head-mounted display removablymountable on a user's head, one or more light sources coupled to thehead-mounted display system and configured to emit light in anirradiated field of view, one or more electromagnetic radiationdetectors coupled to the head-mounted display system and configured toreceive reflected light from a target object irradiated by the one ormore light sources within the irradiated field of view, one or moreprocessors, and one or more computer storage media. The one or morecomputer storage media store instructions that, when executed by the oneor more processors, cause the system to perform operations comprisingdetecting ambient light of a first wavelength within the irradiatedfield of view, emitting light of the first wavelength toward the targetobject, detecting light of the first wavelength reflected by the targetobject, subtracting an intensity of the detected ambient light of thefirst wavelength from an intensity of the detected light reflected bythe target object to calculate a level of light absorption related tothe emitted light and the reflected light from the target object,identifying, based on an absorption database of light absorptionproperties of a plurality of materials, a material characteristic of thetarget object, and displaying, to the user, an output associated withthe material characteristic.

In some embodiments, a wearable spectroscopy system is provided. Thewearable spectroscopy system comprises a head-mounted display removablymountable on a user's head, one or more light sources coupled to thehead-mounted display system and configured to emit light in anirradiated field of view, one or more electromagnetic radiationdetectors coupled to the head-mounted display system and configured toreceive reflected light from a target object irradiated by the one ormore light sources within the irradiated field of view, one or moreprocessors, and one or more computer storage media. The one or morecomputer storage media store instructions that, when executed by the oneor more processors, cause the system to perform operations comprisingemitting light of a first wavelength in an irradiated field of view, thelight comprising a time-encoded variation, detecting light of the firstwavelength reflected from a target object within the irradiated field ofview, identifying, based at least in part on the detected light and thetime-encoded variation, an ambient light component of the detected lightand a reflected component of the detected light, identifying, based atleast in part on the reflected component and an absorption database oflight absorption properties of at least one material, a materialcharacteristic of the target object, and displaying, to the user, anoutput associated with the material characteristic.

Addition examples of embodiments are provide below.

1. A wearable spectroscopy system comprising:

-   -   a head-mounted display system removably mountable on a user's        head;    -   one or more light sources coupled to the head-mounted display        system and configured to emit light in an irradiated field of        view;    -   one or more electromagnetic radiation detectors coupled to the        head-mounted display system and configured to receive reflected        light from a target object irradiated by the one or more light        sources within the irradiated field of view;    -   one or more processors; and    -   one or more computer storage media storing instructions that,        when executed by the one or more processors, cause the system to        perform operations comprising:        -   causing the one or more light sources to emit light;        -   causing the one or more electromagnetic radiation detectors            to detect light from the irradiated field of view including            the target object;        -   determining an ambient light correction by detecting ambient            light levels;        -   applying the ambient light correction to the detected light            to determine levels of light absorption related to the            emitted light and reflected light from the target object;        -   identifying, based on the levels of light absorption, a            characteristic of the target object; and        -   displaying an output associated with the characteristic of            the target object to the user on the head-mounted display            system.

2. The system of example 1, further comprising an absorption database oflight absorption properties of a plurality of materials.

3. The system of example 1, wherein the ambient light correctioncomprises one or more of: an ambient light intensity value, an averageof a plurality of ambient light intensity values, a median of aplurality of ambient light intensity values, and a time-domain ambientlight intensity function.

4. The system of example 1, further comprising at least one eye trackingcamera configured to detect a gaze of the user, wherein the irradiatedfield of view is substantially in the same direction as the detectedgaze.

5. The system of example 1, wherein the one or more electromagneticradiation detectors are further configured to detect the ambient lightlevels.

6. The system of example 1, further comprising an ambient light detectorcoupled to the head-mounted display system and configured to captureambient light not emitted by the one or more light sources, the ambientlight including one or more wavelengths emitted by the one or more lightsources.

7. The system of example 6, wherein the ambient light detector comprisesat least one of a photodiode, a photodetector, and a digital camerasensor.

8. The system of example 6, wherein the instructions, when executed bythe one or more processors, further cause system to perform operationscomprising:

-   -   causing the ambient light detector to detect light while the one        or more light sources are not emitting light; and    -   determining the ambient light correction based at least in part        on the light detected by the ambient light detector.

9. The system of example 1, further comprising an anti-scatter gridcoupled to the head-mounted display system between the target object andthe one or more electromagnetic radiation detectors, the anti-scattergrid aligned to attenuate at least a portion of scattered light andambient light incident upon the anti-scatter grid.

10. The system of example 9, wherein the anti-scatter grid is furtherdisposed between the target object and a detector for detecting ambientlight levels.

11. The system of example 1, wherein the one or more light sources areconfigured to emit the light in a series of time-separated pulses, andwherein the instructions, when executed by the one or more processors,further cause the system to perform operations comprising:

-   -   identifying time-separated pulses of reflected light        corresponding to the time-separated pulses emitted by the one or        more light sources; and    -   determine the ambient light correction based at least in part on        an intensity of light detected at the one or more        electromagnetic radiation detectors between the time-separated        pulses of reflected light.

12. The system of example 11, wherein the time-separated pulses ofreflected light are detected at the one or more electromagneticradiation detectors.

13. The system of example 1, wherein the one or more electromagneticradiation detectors comprises at least one of a photodiode and aphotodetector.

14. The system of example 1, wherein the one or more electromagneticradiation detectors comprises a digital image sensor.

15. The system of example 1, wherein the head-mounted member furthercomprises an inertial measurement unit positional system.

16. The system of example 15, wherein the inertial measurement systemsdetermines a pose orientation of the user's head.

17. The system of example 16, wherein the irradiated field of view is atleast as wide as the pose orientation.

18. The system of example 1, wherein the head-mounted display systemcomprises a waveguide stack configured to output light with selectivelyvariable levels of wavefront divergence.

19. The system of example 18, wherein the waveguide stack compriseswaveguides having optical power.

20. A wearable spectroscopy system comprising:

-   -   a head-mounted display removably mountable on a user's head;    -   one or more light sources coupled to the head-mounted display        system and configured to emit light in an irradiated field of        view;    -   one or more electromagnetic radiation detectors coupled to the        head-mounted display system and configured to receive reflected        light from a target object irradiated by the one or more light        sources within the irradiated field of view;    -   an anti-scatter grid disposed between the electromagnetic        radiation detector and the target object, the anti-scatter grid        configured to attenuate at least one of scattered light and        ambient light incident thereon;    -   one or more processors; and    -   one or more computer storage media storing instructions that,        when executed by the one or more processors, cause the system to        perform operations comprising:        -   emitting, from the one or more light sources, light of a            first wavelength in an irradiated field of view;        -   detecting, at the one or more electromagnetic radiation            detectors, light of the first wavelength reflected from a            target object within the irradiated field of view;        -   identifying, based on an absorption database of light            absorption properties of at least one material, a material            characteristic of the target object; and        -   causing a graphics processor unit to display, to the user,            an output associated with the material characteristic.

21. The system of example 20, wherein the instructions, when executed bythe one or more processors, further cause the system to performoperations comprising:

-   -   detecting an intensity of ambient light at the first wavelength;        and    -   determining an ambient light-corrected intensity of reflected        light by subtracting the intensity of ambient light from an        intensity of the light of the first wavelength detected at the        electromagnetic radiation detector,    -   wherein the material characteristic of the target object is        identified based at least in part on the ambient light-corrected        intensity.

22. The system of example 21, wherein the intensity of ambient light atthe first wavelength is detected while the light source is not emittinglight.

23. The system of example 20, wherein the instructions, when executed bythe one or more processors, further cause the system to performoperations comprising:

-   -   detecting a plurality of intensities of ambient light at the        first wavelength; and    -   determining an ambient light-corrected intensity of reflected        light by subtracting, from an intensity of the light detected at        the electromagnetic radiation detector, one of: an average of        the plurality of intensities of ambient light, a median of the        plurality of intensities of ambient light, and a time-domain        ambient light intensity function corresponding to the plurality        of intensities of ambient light,    -   wherein the material characteristic of the target object is        identified based at least in part on the ambient light-corrected        intensity.

24. The system of example 23, wherein the plurality of intensities ofambient light at the first wavelength are detected while the lightsource is not emitting light.

25. The system of example 20, wherein the light of the first wavelengthis emitted in a series of time-separated pulses, and wherein theinstructions, when executed by the one or more processors, further causethe system to perform operations comprising:

-   -   identifying an ambient light component of the detected light        based on recognizing the time-separated pulses in the detected        light; and    -   determining an ambient light-corrected intensity of reflected        light by subtracting the ambient light component from an        intensity of the time-separated pulses in the detected light,    -   wherein the material characteristic of the target object is        identified based at least in part on the ambient light-corrected        intensity.

26. A wearable spectroscopy system comprising:

-   -   a head-mounted display removably mountable on a user's head;    -   one or more light sources coupled to the head-mounted display        system and configured to emit light in an irradiated field of        view;    -   one or more electromagnetic radiation detectors coupled to the        head-mounted display system and configured to receive reflected        light from a target object irradiated by the one or more light        sources within the irradiated field of view;    -   one or more processors; and    -   one or more computer storage media storing instructions that,        when executed by the one or more processors, cause the system to        perform operations comprising:        -   detecting ambient light of a first wavelength within the            irradiated field of view;        -   emitting light of the first wavelength toward the target            object;        -   detecting light of the first wavelength reflected by the            target object;        -   subtracting an intensity of the detected ambient light of            the first wavelength from an intensity of the detected light            reflected by the target object to calculate a level of light            absorption related to the emitted light and the reflected            light from the target object;        -   identifying, based on an absorption database of light            absorption properties of a plurality of materials, a            material characteristic of the target object; and        -   displaying, to the user, an output associated with the            material characteristic.

27. The system of example 26, wherein the ambient light is detected at atime when the light source is not emitting light.

28. The system of example 26, wherein detecting the ambient lightcomprises detecting an intensity of the ambient light at a plurality oftimes and calculating an average ambient light intensity or a medianambient light intensity.

29. The system of example 26, wherein detecting the ambient lightcomprises detecting an intensity of the ambient light at a plurality oftimes and calculating a time-domain ambient light intensity function,and wherein the intensity of ambient light subtracted from the intensityof the detected light is determined based at least in part on thetime-domain ambient light intensity function.

30. The system of example 26, further comprising an ambient lightdetector, the ambient light detector comprising one or more of aphotodiode, a photodetector, and a digital camera sensor.

31. A wearable spectroscopy system comprising:

-   -   a head-mounted display removably mountable on a user's head;    -   one or more light sources coupled to the head-mounted display        system and configured to emit light in an irradiated field of        view;    -   one or more electromagnetic radiation detectors coupled to the        head-mounted display system and configured to receive reflected        light from a target object irradiated by the one or more light        sources within the irradiated field of view;    -   one or more processors; and    -   one or more computer storage media storing instructions that,        when executed by the one or more processors, cause the system to        perform operations comprising:        -   emitting light of a first wavelength in an irradiated field            of view, the light comprising a time-encoded variation;        -   detecting light of the first wavelength reflected from a            target object within the irradiated field of view;        -   identifying, based at least in part on the detected light            and the time-encoded variation, an ambient light component            of the detected light and a reflected component of the            detected light;        -   identifying, based at least in part on the reflected            component and an absorption database of light absorption            properties of at least one material, a material            characteristic of the target object; and        -   displaying, to the user, an output associated with the            material characteristic.

32. The system of example 31, wherein the time-encoded variationcomprises a plurality of time-separated pulses of the light of the firstwavelength, and wherein identifying the ambient light component and thereflected component comprises identifying time-separated pulses in thedetected light corresponding to the time-separated pulses of the emittedlight.

33. The system of example 32, wherein identifying the ambient lightcomponent and the reflected component further comprises:

-   -   measuring an intensity of detected light at a time between two        of the time-separated pulses in the detected light; and    -   subtracting the measured intensity from a maximum intensity of        one or more of the time-separated pulses.

34. The system of example 31, wherein the time-encoded variationcomprises at least one of frequency modulation and amplitude modulation.

35. The system of example 31, wherein the time-encoded variationcomprises at least one of a Manchester code, a Hamming code, aheterodyne signal, and a pseudo-random intensity variation.

These and many other features and advantages of the present inventionwill be appreciated when the following figures and description arefurther taken into account.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates certain aspects of an augmented reality systempresentation to a user.

FIG. 1.1 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIGS. 1.2 to 1.4 illustrate relationships between radius of curvatureand focal radius.

FIG. 1.5 illustrates a representation of the accommodation-vergenceresponse of the human visual system.

FIG. 1.6 illustrates examples of different accommodative states andvergence states of a pair of eyes of the user.

FIG. 1.7 illustrates an example of a representation of a top-down viewof a user viewing content via a display system.

FIG. 1.8 illustrates another example of a representation of a top-downview of a user viewing content via a display system.

FIG. 1.9 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence.

FIG. 1.10 illustrates an example of a waveguide stack for outputtingimage information to a user.

FIG. 1.11 illustrates an example of exit beams outputted by a waveguide.

FIG. 1.12 illustrates an example of a stacked waveguide assembly inwhich each depth plane includes images formed using multiple differentcomponent colors.

FIG. 1.13 illustrates a cross-sectional side view of an example of a setof stacked waveguides that each includes an incoupling optical element.

FIG. 1.14 illustrates a perspective view of an example of the pluralityof stacked waveguides of FIG. 1.13.

FIG. 1.15 illustrates a top-down plan view of an example of theplurality of stacked waveguides of FIGS. 1.13 and 1.14.

FIGS. 2A-2D illustrate certain aspects of various augmented realitysystems for wearable computing applications, featuring a head-mountedcomponent operatively coupled to local and remote process and datacomponents.

FIG. 3 illustrates certain aspects of a connectivity paradigm between awearable augmented or virtual reality system and certain remoteprocessing and/or data storage resources.

FIGS. 4A-4D illustrate various aspects of pulse oximetry configurationsand calibration curves related to scattering of light in oxygenation ofblood.

FIG. 5 illustrates a head-mounted spectroscopy system integrating AR/VRfunctionality according to some embodiments,

FIG. 6 illustrates various aspects of a wearable AR/VR system featuringintegrated spectroscopy modules according to some embodiments.

FIGS. 7A-7B are an example light saturation curve chart indicative ofselect properties by wavelengths.

FIG. 8 illustrates a method for identifying materials or materialproperties through a head-mounted spectroscopy system according to someembodiments.

FIGS. 9A-9C are example graphs illustrating the effect of ambient lighton spectroscopic measurements in some embodiments.

FIGS. 10A-10C are graphs illustrating an example process of correctingfor ambient light using time-domain multiplexing according to someembodiments.

FIG. 11 illustrates a method for identifying materials or materialproperties through a head-mounted spectroscopy system with correctionfor ambient light according to some embodiments.

DETAILED DESCRIPTION

By virtue of the fact that at least some of the components in a wearablecomputing system, such as an AR or VR system, are close to the body ofthe user operating them, there is an opportunity to utilize some ofthese system components to conduct certain physiologic monitoringrelative to the user and to perform such monitoring spontaneously, asdesired. For example, physiologic monitoring may be conducted bymeasuring light absorption.

In conventional light absorption measurement techniques (for examplepulse oximetry meters attachable to a person's finger as in FIG. 4A orin glucose detection), light is emitted in a controlled and fixeddirection and received in a controlled and fixed receiver. Light ispulsed at different wavelengths through surrounding tissue structureswhile also being detected at another side of the tissue structure (andtherefore measuring light properties such as absorption and scatter). Insuch systems, the measurement of light emitted compared to themeasurement of light detected can provide an output that is proportionalto, or reads as, an estimated tissue or tissue property (for example, anestimated blood oxygen saturation level for pulse oximetry meters), orsimply a material or tissue type otherwise. Calibration curves depictinga ratio of light of interest relative to other light are also possibleto predict properties of underlying tissue as a function of the lightincident to it as shown in FIG. 4D.

Raman spectroscopy is another technique that measures inelasticscattering of photons released by irradiated molecules. Specificmolecules will present specific shifts of wavelengths when irradiated,thereby presenting unique scattering effects that may be used to measureand quantify molecules within a sample.

FIG. 4B illustrates a chart of the absorption spectra of hemoglobin thatis oxygenated (806) versus deoxygenated (808), and as shown in suchplots (806, 808), in the red light wavelength range of theelectromagnetic spectrum, such as around 660 nm, there is a notabledifference in absorption for oxygenated versus deoxygenated hemoglobin,whereas there is an inverted difference at around 940 nm in the infraredwavelength range. The plots of the absorption spectra of oxygenated(806) and deoxygenated (808) hemoglobin cross within the near infraredregion, which spans, e.g., between approximately 700 nm and 900 nm.Pulsing radiation at such wavelengths and detecting with a pulseoximeter is known to take advantage of such absorption differences inthe determination of oxygen saturation for a particular user.

While pulse oximeters (802) typically are configured to at leastpartially encapsulate a tissue structure such as a finger (804) or earlobe, certain desktop style systems have been suggested, such as that(812) depicted in FIG. 4C, to observe absorption differences in vesselsof the eye, such as retinal vessels, but may be configured to detectproperties of other tissues as well.

Such a configuration (812) may be termed a flow oximeter or spectroscopesystem and may comprise components as shown, including a camera (816),zoom lens (822), first (818) and second (819) light emitting diodes(LEDs), and one or more beam splitters (814). While it would be valuableto certain users, such as high-altitude hikers, athletes, or personswith certain cardiovascular or respiratory problems, to be able toretrieve information of their blood oxygen saturation as they move abouttheir day and conduct their activities, or for caregivers to analyzetissue in real time for underlying abnormalities, most configurationsinvolve a somewhat inconvenient encapsulation of a tissue structure, orare not portable or wearable, do not consider other absorptionproperties indicative of other tissue states or materials, or do notcorrelate gaze a user is looking at as part of directionality of itssensors (in other words, selectivity of target objects of foridentification and analysis by spectroscopy is lacking).

Advantageously, in some embodiments, a solution is presented hereinwhich combines the convenience of wearable computing in the form of anAR or VR display system with an imaging means to determine additionaltissue identification and properties in real time within a field of viewof a user. In addition, the accuracy of tissue identification may beincreased by accounting for ambient light, as disclosed herein.

In some embodiments, a mixed reality system is configured to performspectroscopy. Mixed reality (alternatively abbreviated as “MR”)typically involves virtual objects integrated into and responsive to thenatural world. For example, in an MR scenario, AR content may beoccluded by real world objects and/or be perceived as interacting withother objects (virtual or real) in the real world. Throughout thisdisclosure, reference to AR, VR or MR is not limiting on the inventionand the techniques may be applied to any context.

Some embodiments are directed to a wearable system for identifyingsubstances (such as tissue, cells within tissue, or properties withincells/tissue) as a function of light wavelength emitted from a lightemitter and subsequently received by, reflected to, and detected at oneor more electromagnetic radiation detectors forming part of head-mountedmember removably coupleable, or mountable, to a user's head. Though thisdisclosure mainly references tissue, or tissue properties, as a subjectfor analysis according to various embodiments, the technologies andtechniques and components are not limited to such. Some embodimentsutilize one or more light sources, such as electromagnetic radiationemitters coupled to the head-mounted frame, to emit light in one or morewavelengths in a user-selected direction. Such embodiments permitcontinuous, and even passive, measurements. For example, a user wearinga head mounted system could conduct a given activity, but inward facingsensors could detect properties of the eye without interfering with theactivity.

It will be appreciated that the presence of ambient light in theenvironment may complicate these spectroscopic systems and methods. Inthe presence of ambient light (e.g., light present in the ambientenvironment but not outputted by the wearable system for purposes ofsubstance identification), the light received by the electromagneticradiation detectors may include a combination of the reflected emittedlight and ambient light. Because the various properties (including theamount) of the ambient light may not be known or predicted, thecontribution of the ambient light to the light detected by the systemmay yield accurate spectroscopic data. In some embodiments, the wearablesystem is configured to account for the effects of ambient light on thespectroscopic methods described herein. The wearable system may beconfigured to emit light from a light source of the system, detect aportion of the emitted light reflected from a surface of a targetobject, apply an ambient light correction, and identify one or morematerial properties of the object based on properties of the reflectedlight, such as absorption at a wavelength or range of wavelengths.

Accordingly, the controller, light source(s), and/or electromagneticradiation detector(s) may further be configured to reduce and/or removethe confounding effect of ambient light contributions to the detectedlight. In some embodiments, the system may be configured to detect abaseline or ambient light correction that can be subtracted fromspectroscopic measurements. The baseline or ambient light correction mayinclude, for example, one or more of an ambient light intensity value,an average, median or other statistical quantity derived from aplurality of ambient light intensity values, a time-domain ambient lightintensity function determined based on one or more measured ambientlight intensity values, etc. For example, a baseline or ambient lightcorrection may be obtained by detecting light at a radiation detector(which may be used for spectroscopic analysis) while light is not beingemitted for spectroscopic measurement. In another example, the systemmay include an ambient light sensor separate from the photodetector orother radiation detectors used for spectroscopic analysis. In anotherexample, the system may utilize time-domain multiplexing in the emittedlight signal to remove the ambient light contribution, with or withoutusing a separate ambient light sensor.

In some embodiments, an anti-scatter grid may be provided an opticalpath between an object being analyzed in the radiation detectors used tomeasure reflected light to determine absorbance. The anti-scatter gridprevents scattered light from being captured by the radiation detectors.In some embodiments, the scattered light may be understood to be ambientlight and, as such, is desirably excluded from the reflected lightmeasurement.

Advantageously, the wearable system may be configured to identify and/ormeasure properties of objects on or part of the user, or separate fromthe user. For example, a user could wear a system configured to lookinward to the user's eyes and identify or measure tissue properties ofthe eye, such as blood concentration in a blood vessel of the eye. Inother examples of inward systems, fluids such as intraocular fluid maybe analyzed and not simply tissue properties. In other examples, asystem could comprise sensors that look outward towards the externalworld and identify or measure tissue or material properties other thanthe eye, such as an extremity of the user or object in the ambientenvironment apart from the user.

In outward looking systems, eye tracking cameras coupled to thehead-mounted member can determine the directional gaze a user islooking, and a processor or controller may correlate that gaze withobservation of a real world target object through images captured from areal-world capturing system (such as cameras or depth sensors) coupledto the head-mounted member. Light sources coupled to the head-mountedsystem emit light away from the user, such as infrared light for examplefrom an electromagnetic radiation emitter, and in some embodiments emitlight to create an irradiation pattern in a substantially same directionas a gaze direction determined by the eye tracking cameras, therebyemitting upon the target object.

In some embodiments, real world capturing systems capture an object. Forexample a depth sensor, such as a vertical cavity surface emittinglaser, may determine the outline of an object through collecting time offlight signals impacting the object. The object, once identified at itscontours by such real-world capturing system may be highlighted andavailable for labeling. In some embodiments, a camera system of a givenfield of view defines an area available for highlighting and labelling.For example, a camera correlating to a user's gaze may encompass a 5degree field of view, 10 degree field of view, or suitable incrementspreferably up to a 30 degree central vision field of view that the lightsource will emit light substantially within.

In some embodiments, such a system further comprises one or moreelectromagnetic radiation detectors or photodetectors coupled to thehead-mounted member configured to receive reflected light that wasemitted from the light source and reflected from the target object; anda controller operatively coupled to the one or more electromagneticradiation emitters and one or more electromagnetic radiation detectorsconfigured to cause the one or more electromagnetic radiation emittersto emit pulses of light while also causing the one or moreelectromagnetic radiation detectors to detect levels of light absorptionrelated to the emitted pulses of light as a function of any receivedreflected light of a particular pulse emission.

In some embodiments, the system further comprises a processor to match awavelength of reflected light received by a detector from the targetobject to a characteristic such as a particular material, tissue type,or property (e.g., a change in one or more chemical properties orcompositions of a tissue) of an underlying tissue. In some embodiments,other light characteristics are determined, such as polarization changesrelative to emitted light and detected light or scattering effects,though for purposes of this description wavelength characteristics areused as an exemplary light characteristic. For example, in someembodiments, an inward electromagnetic radiation emitter emits light inthe infrared spectrum to the retina of a user, receives reflected light,and matches the wavelength of the reflected light to determine aphysical property such as the type of tissue or oxygen saturation in thetissue. In some embodiments, the system comprises outward facing lightsources, and emits infrared light to a target object (such as anextremity of a user or third person), receives reflected light, andmatches the reflected light wavelength to determine the observedmaterial. For example, such an outward facing system may detect thepresence of cancerous cells among healthy cells. Because cancerous, orother abnormal cells, reflect and absorb light differently than healthycells, a reflection of light at certain wavelengths can indicate thepresence and amount of abnormality.

In some embodiments, the controller receives the captured target objectfrom the real world capturing system, and applies a label to the targetobject indicative of the identified property. In some embodiments, thelabel is a textual label or prompt within a display of the headmounted-member. In some embodiments, the label is an audio prompt to auser. In some embodiments, the label is a virtual image of similartissue, such as referenced in a medical book, superimposed near thetarget object for ready comparative analysis by the user.

In some embodiments, the head-mounted member may comprise an eyeglassesframe. The eyeglasses frame may be a binocular eyeglasses frame. The oneor more radiation emitters may comprise a light source, such as a lightemitting diode. The one or more radiation emitters may comprise aplurality of light sources configured to emit electromagnetic radiationat two or more different wavelengths. The plurality of light sources maybe configured to emit electromagnetic radiation at a first wavelength ofabout 660 nanometers, and a second wavelength of about 940 nanometers.The one or more radiation emitters may be configured to emitelectromagnetic radiation at the two different wavelengths sequentially.The one or more radiation emitters may be configured to emitelectromagnetic radiation at the two predetermined wavelengthssimultaneously. The one or more electromagnetic radiation detectors maycomprise a device selected from the group consisting of: a photodiode, aphotodetector, and a digital camera sensor. The one or moreelectromagnetic radiation detectors may be positioned and oriented toreceive light reflected after encountering a target object. The one ormore electromagnetic radiation detectors may be positioned and orientedto receive light reflected after encountering observed tissue ormaterial; that is, the one or more electromagnetic radiation detectorsare oriented substantially in the same direction as the one or moreelectromagnetic radiation emitters, whether inward facing towards auser's eye or outward facing towards a user's environment.

The controller may be further configured to cause the plurality of lightsources to emit a cyclic pattern of first wavelength on, then secondwavelength on, then both wavelengths off, such that the one or moreelectromagnetic radiation detectors detect the first and secondwavelengths separately. The controller may be configured to cause theplurality of light emitting diodes to emit a cyclic pattern of firstwavelength on, then second wavelength on, then both wavelengths off, ina cyclic pulsing pattern about thirty times per second.

In some embodiments, the controller may be configured to calculate aratio of first wavelength light measurement to second wavelength lightmeasurement. In some embodiments this ratio may be further converted toan oxygen saturation reading via a lookup table based at least in partupon the Beer-Lambert law. In some embodiments, the ratio is convertedto a material identifier in external lookup tables, such as stored in anabsorption database module on a head-mounted member or coupled to ahead-mounted member on a local or remote processing module. For example,an absorption database module for absorption ratios or wavelengthreflection of particular tissues may be stored in a “cloud” storagesystem accessible by health care providers and accessed through a remoteprocessing module. In some embodiments, an absorption database modulemay store absorption properties (such as wavelength ratios or wavelengthreflections) for certain foods and be permanently stored on a localprocessing module to the head-mounted member.

In this way, the controller may be configured to operate the one or moreelectromagnetic radiation emitters and one or more electromagneticradiation detectors to function as a broad use head-mountedspectroscope. The controller may be operatively coupled to an opticalelement coupled to the head-mounted member and viewable by the user,such that the output of the controller indicating the wavelengthproperties indicative of a particular tissue property or materialotherwise may be viewed by the user through the optical element. The oneor more electromagnetic radiation detectors may comprise a digital imagesensor comprising a plurality of pixels, wherein the controller isconfigured to automatically detect a subset of pixels which arereceiving the light reflected after encountering, for example, tissue orcells within the tissue. In some embodiments, such subset of pixels areused to produce an output representative of the target object within thefield of view of the digital image sensor. For example, the output maybe a display label that is indicative of an absorption level of thetissue. In some embodiments, comparative values are displayed as anoutput. For example, an output may be a percentage saturation of oxygenof blood from a first analysis time and a percentage saturation ofoxygen at a second analysis time with a rate of change noted between thetwo times. In these embodiments, ailments such as diabetic retinopathymay be detected by recognizing changes in measured properties over time.

In some embodiments, the controller may be configured to automaticallydetect the subset of pixels based at least in part upon reflected lightluminance differences amongst signals associated with the pixels. Thecontroller may be configured to automatically detect the subset ofpixels based at least in part upon reflected light absorptiondifferences amongst signals associated with the pixels. In suchembodiments, such subsets may be isolated pixels and flagged for furtheranalysis, such as additional irradiation or mapping, or a virtual imagemay be overlaid on such pixels to provide visual contrast to theisolated pixels displaying other properties to serve as a notice to auser of the different properties of the subpixels identified by thesystem.

In some embodiments, the system data collection is time multiplexed notonly for pulsing and recording light pulses, but passively collected atmultiple times a day. In some embodiments, a GPS or other similarmapping system is coupled to the system to correlate a user's locationor time of day with certain physiological data collected. For example, auser may track physiological responses relative to certain locations oractivities throughout a day.

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout. Unless specifically indicatedotherwise, the drawings are schematic not necessarily drawn to scale.

FIG. 1.1 illustrates a conventional display system for simulatingthree-dimensional imagery for a user. It will be appreciated that auser's eyes are spaced apart and that, when looking at a real object inspace, each eye will have a slightly different view of the object andmay form an image of the object at different locations on the retina ofeach eye. This may be referred to as binocular disparity and may beutilized by the human visual system to provide a perception of depth.Conventional display systems simulate binocular disparity by presentingtwo distinct images 190, 200 with slightly different views of the samevirtual object—one for each eye 210, 220—corresponding to the views ofthe virtual object that would be seen by each eye were the virtualobject a real object at a desired depth. These images provide binocularcues that the user's visual system may interpret to derive a perceptionof depth.

With continued reference to FIG. 1.1, the images 190, 200 are spacedfrom the eyes 210, 220 by a distance 230 on a z-axis. The z-axis isparallel to the optical axis of the viewer with their eyes fixated on anobject at optical infinity directly ahead of the viewer. The images 190,200 are flat and at a fixed distance from the eyes 210, 220. Based onthe slightly different views of a virtual object in the images presentedto the eyes 210, 220, respectively, the eyes may naturally rotate suchthat an image of the object falls on corresponding points on the retinasof each of the eyes, to maintain single binocular vision. This rotationmay cause the lines of sight of each of the eyes 210, 220 to convergeonto a point in space at which the virtual object is perceived to bepresent. As a result, providing three-dimensional imagery conventionallyinvolves providing binocular cues that may manipulate the vergence ofthe user's eyes 210, 220, and that the human visual system interprets toprovide a perception of depth.

Generating a realistic and comfortable perception of depth ischallenging, however. It will be appreciated that light from objects atdifferent distances from the eyes have wavefronts with different amountsof divergence. FIGS. 1.2 to 1.4 illustrate relationships betweendistance and the divergence of light rays. The distance between theobject and the eye 210 is represented by, in order of decreasingdistance, R1, R2, and R3. As shown in FIGS. 1.2 to 1.4, the light raysbecome more divergent as distance to the object decreases. Conversely,as distance increases, the light rays become more collimated. Statedanother way, it may be said that the light field produced by a point(the object or a part of the object) has a spherical wavefrontcurvature, which is a function of how far away the point is from the eyeof the user. The curvature increases with decreasing distance betweenthe object and the eye 210. While only a single eye 210 is illustratedfor clarity of illustration in FIGS. 1.2 to 1.4 and other figuresherein, the discussions regarding eye 210 may be applied to both eyes210 and 220 of a viewer.

With continued reference to FIGS. 1.2 to 1.4, light from an object thatthe viewer's eyes are fixated on may have different degrees of wavefrontdivergence. Due to the different amounts of wavefront divergence, thelight may be focused differently by the lens of the eye, which in turnmay require the lens to assume different shapes to form a focused imageon the retina of the eye. Where a focused image is not formed on theretina, the resulting retinal blur acts as a cue to accommodation thatcauses a change in the shape of the lens of the eye until a focusedimage is formed on the retina. For example, the cue to accommodation maytrigger the ciliary muscles surrounding the lens of the eye to relax orcontract, thereby modulating the force applied to the suspensoryligaments holding the lens, thus causing the shape of the lens of theeye to change until retinal blur of an object of fixation is eliminatedor minimized, thereby forming a focused image of the object of fixationon the retina (e.g., fovea) of the eye. The process by which the lens ofthe eye changes shape may be referred to as accommodation, and the shapeof the lens of the eye required to form a focused image of the object offixation on the retina (e.g., fovea) of the eye may be referred to as anaccommodative state.

With reference now to FIG. 1.5, a representation of theaccommodation-vergence response of the human visual system isillustrated. The movement of the eyes to fixate on an object causes theeyes to receive light from the object, with the light forming an imageon each of the retinas of the eyes. The presence of retinal blur in theimage formed on the retina may provide a cue to accommodation, and therelative locations of the image on the retinas may provide a cue tovergence. The cue to accommodation causes accommodation to occur,resulting in the lenses of the eyes each assuming a particularaccommodative state that forms a focused image of the object on theretina (e.g., fovea) of the eye. On the other hand, the cue to vergencecauses vergence movements (rotation of the eyes) to occur such that theimages formed on each retina of each eye are at corresponding retinalpoints that maintain single binocular vision. In these positions, theeyes may be said to have assumed a particular vergence state. Withcontinued reference to FIG. 1.5, accommodation may be understood to bethe process by which the eye achieves a particular accommodative state,and vergence may be understood to be the process by which the eyeachieves a particular vergence state. As indicated in FIG. 1.5, theaccommodative and vergence states of the eyes may change if the userfixates on another object. For example, the accommodated state maychange if the user fixates on a new object at a different depth on thez-axis.

Without being limited by theory, it is believed that viewers of anobject may perceive the object as being “three-dimensional” due to acombination of vergence and accommodation. As noted above, vergencemovements (e.g., rotation of the eyes so that the pupils move toward oraway from each other to converge the lines of sight of the eyes tofixate upon an object) of the two eyes relative to each other areclosely associated with accommodation of the lenses of the eyes. Undernormal conditions, changing the shapes of the lenses of the eyes tochange focus from one object to another object at a different distancewill automatically cause a matching change in vergence to the samedistance, under a relationship known as the “accommodation-vergencereflex.” Likewise, a change in vergence will trigger a matching changein lens shape under normal conditions.

With reference now to FIG. 1.6, examples of different accommodative andvergence states of the eyes are illustrated. The pair of eyes 222 a isfixated on an object at optical infinity, while the pair eyes 222 b arefixated on an object 221 at less than optical infinity. Notably, thevergence states of each pair of eyes is different, with the pair of eyes222 a directed straight ahead, while the pair of eyes 222 converge onthe object 221. The accommodative states of the eyes forming each pairof eyes 222 a and 222 b are also different, as represented by thedifferent shapes of the lenses 210 a, 220 a.

Undesirably, many users of conventional “3-D” display systems find suchconventional systems to be uncomfortable or may not perceive a sense ofdepth at all due to a mismatch between accommodative and vergence statesin these displays. As noted above, many stereoscopic or “3-D” displaysystems display a scene by providing slightly different images to eacheye. Such systems are uncomfortable for many viewers, since they, amongother things, simply provide different presentations of a scene andcause changes in the vergence states of the eyes, but without acorresponding change in the accommodative states of those eyes. Rather,the images are shown by a display at a fixed distance from the eyes,such that the eyes view all the image information at a singleaccommodative state. Such an arrangement works against the“accommodation-vergence reflex” by causing changes in the vergence statewithout a matching change in the accommodative state. This mismatch isbelieved to cause viewer discomfort. Display systems that provide abetter match between accommodation and vergence may form more realisticand comfortable simulations of three-dimensional imagery.

Without being limited by theory, it is believed that the human eyetypically may interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited numbers of depthplanes. In some embodiments, the different presentations may provideboth cues to vergence and matching cues to accommodation, therebyproviding physiologically correct accommodation-vergence matching.

With continued reference to FIG. 1.6, two depth planes 240,corresponding to different distances in space from the eyes 210, 220,are illustrated. For a given depth plane 240, vergence cues may beprovided by the displaying of images of appropriately differentperspectives for each eye 210, 220. In addition, for a given depth plane240, light forming the images provided to each eye 210, 220 may have awavefront divergence corresponding to a light field produced by a pointat the distance of that depth plane 240.

In the illustrated embodiment, the distance, along the z-axis, of thedepth plane 240 containing the point 221 is 1 m. As used herein,distances or depths along the z-axis may be measured with a zero-pointlocated at the pupils of the user's eyes. Thus, a depth plane 240located at a depth of 1 m corresponds to a distance of 1 m away from thepupils of the user's eyes, on the optical axis of those eyes with theeyes directed towards optical infinity. As an approximation, the depthor distance along the z-axis may be measured from the display in frontof the user's eyes (e.g., from the surface of a waveguide), plus a valuefor the distance between the device and the pupils of the user's eyes.That value may be called the eye relief and corresponds to the distancebetween the pupil of the user's eye and the display worn by the user infront of the eye. In practice, the value for the eye relief may be anormalized value used generally for all viewers. For example, the eyerelief may be assumed to be 20 mm and a depth plane that is at a depthof 1 m may be at a distance of 980 mm in front of the display.

With reference now to FIGS. 1.7 and 1.8, examples of matchedaccommodation-vergence distances and mismatched accommodation-vergencedistances are illustrated, respectively. As illustrated in FIG. 1.7, thedisplay system may provide images of a virtual object to each eye 210,220. The images may cause the eyes 210, 220 to assume a vergence statein which the eyes converge on a point 15 on a depth plane 240. Inaddition, the images may be formed by a light having a wavefrontcurvature corresponding to real objects at that depth plane 240. As aresult, the eyes 210, 220 assume an accommodative state in which theimages are in focus on the retinas of those eyes. Thus, the user mayperceive the virtual object as being at the point 15 on the depth plane240.

It will be appreciated that each of the accommodative and vergencestates of the eyes 210, 220 are associated with a particular distance onthe z-axis. For example, an object at a particular distance from theeyes 210, 220 causes those eyes to assume particular accommodativestates based upon the distances of the object. The distance associatedwith a particular accommodative state may be referred to as theaccommodation distance, A_(d). Similarly, there are particular vergencedistances, V_(d), associated with the eyes in particular vergencestates, or positions relative to one another. Where the accommodationdistance and the vergence distance match, the relationship betweenaccommodation and vergence may be said to be physiologically correct.This is considered to be the most comfortable scenario for a viewer.

In stereoscopic displays, however, the accommodation distance and thevergence distance may not always match. For example, as illustrated inFIG. 1.8, images displayed to the eyes 210, 220 may be displayed withwavefront divergence corresponding to depth plane 240, and the eyes 210,220 may assume a particular accommodative state in which the points 15a, 15 b on that depth plane are in focus. However, the images displayedto the eyes 210, 220 may provide cues for vergence that cause the eyes210, 220 to converge on a point 15 that is not located on the depthplane 240. As a result, the accommodation distance corresponds to thedistance from the pupils of the eyes 210, 220 to the depth plane 240,while the vergence distance corresponds to the larger distance from thepupils of the eyes 210, 220 to the point 15, in some embodiments. Theaccommodation distance is different from the vergence distance.Consequently, there is an accommodation-vergence mismatch. Such amismatch is considered undesirable and may cause discomfort in the user.It will be appreciated that the mismatch corresponds to distance (e.g.,V_(d)−A_(d)) and may be characterized using diopters.

In some embodiments, it will be appreciated that a reference point otherthan pupils of the eyes 210, 220 may be utilized for determiningdistance for determining accommodation-vergence mismatch, so long as thesame reference point is utilized for the accommodation distance and thevergence distance. For example, the distances could be measured from thecornea to the depth plane, from the retina to the depth plane, from theeyepiece (e.g., a waveguide of the display device) to the depth plane,from the center of rotation of an eye, and so on.

Without being limited by theory, it is believed that users may stillperceive accommodation-vergence mismatches of up to about 0.25 diopter,up to about 0.33 diopter, and up to about 0.5 diopter as beingphysiologically correct, without the mismatch itself causing significantdiscomfort. In some embodiments, display systems disclosed herein (e.g.,the display system 250, FIG. 1.10) present images to the viewer havingaccommodation-vergence mismatch of about 0.5 diopter or less. In someother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.33 diopter or less. In yetother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.25 diopter or less, includingabout 0.1 diopter or less.

FIG. 1.9 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence. The displaysystem includes a waveguide 270 that is configured to receive light 770that is encoded with image information, and to output that light to theuser's eye 210. The waveguide 270 may output the light 650 with adefined amount of wavefront divergence corresponding to the wavefrontdivergence of a light field produced by a point on a desired depth plane240. In some embodiments, the same amount of wavefront divergence isprovided for all objects presented on that depth plane. In addition, itwill be illustrated that the other eye of the user may be provided withimage information from a similar waveguide.

In some embodiments, a single waveguide may be configured to outputlight with a set amount of wavefront divergence corresponding to asingle or limited number of depth planes and/or the waveguide may beconfigured to output light of a limited range of wavelengths.Consequently, in some embodiments, a plurality or stack of waveguidesmay be utilized to provide different amounts of wavefront divergence fordifferent depth planes and/or to output light of different ranges ofwavelengths. As used herein, it will be appreciated that a depth planemay follow the contours of a flat or a curved surface. In someembodiments, advantageously for simplicity, the depth planes may followthe contours of flat surfaces.

FIG. 1.10 illustrates an example of a waveguide stack for outputtingimage information to a user. A display system 250 includes a stack ofwaveguides, or stacked waveguide assembly, 260 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 270, 280, 290, 300, 310. It will be appreciated that thedisplay system 250 may be considered a light field display in someembodiments. In addition, the waveguide assembly 260 may also bereferred to as an eyepiece.

In some embodiments, the display system 250 may be configured to providesubstantially continuous cues to vergence and multiple discrete cues toaccommodation. The cues to vergence may be provided by displayingdifferent images to each of the eyes of the user, and the cues toaccommodation may be provided by outputting the light that forms theimages with selectable discrete amounts of wavefront divergence. Statedanother way, the display system 250 may be configured to output lightwith variable levels of wavefront divergence. In some embodiments, eachdiscrete level of wavefront divergence corresponds to a particular depthplane and may be provided by a particular one of the waveguides 270,280, 290, 300, 310.

With continued reference to FIG. 1.10, the waveguide assembly 260 mayalso include a plurality of features 320, 330, 340, 350 between thewaveguides. In some embodiments, the features 320, 330, 340, 350 may beone or more lenses. The waveguides 270, 280, 290, 300, 310 and/or theplurality of lenses 320, 330, 340, 350 may be configured to send imageinformation to the eye with various levels of wavefront curvature orlight ray divergence. Each waveguide level may be associated with aparticular depth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 360, 370,380, 390, 400 may function as a source of light for the waveguides andmay be utilized to inject image information into the waveguides 270,280, 290, 300, 310, each of which may be configured, as describedherein, to distribute incoming light across each respective waveguide,for output toward the eye 210. Light exits an output surface 410, 420,430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 andis injected into a corresponding input surface 460, 470, 480, 490, 500of the waveguides 270, 280, 290, 300, 310. In some embodiments, each ofthe input surfaces 460, 470, 480, 490, 500 may be an edge of acorresponding waveguide, or may be part of a major surface of thecorresponding waveguide (that is, one of the waveguide surfaces directlyfacing the world 510 or the viewer's eye 210). It will be appreciatedthat the major surfaces of a waveguide correspond to the relativelylarge area surfaces of the waveguide between which the thickness of thewaveguide extends. In some embodiments, a single beam of light (e.g. acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 210 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide. In some embodiments, asingle one of the image injection devices 360, 370, 380, 390, 400 may beassociated with and inject light into a plurality (e.g., three) of thewaveguides 270, 280, 290, 300, 310.

In some embodiments, the image injection devices 360, 370, 380, 390, 400are discrete displays that each produce image information for injectioninto a corresponding waveguide 270, 280, 290, 300, 310, respectively. Insome other embodiments, the image injection devices 360, 370, 380, 390,400 are the output ends of a single multiplexed display which may, e.g.,pipe image information via one or more optical conduits (such as fiberoptic cables) to each of the image injection devices 360, 370, 380, 390,400. It will be appreciated that the image information provided by theimage injection devices 360, 370, 380, 390, 400 may include light ofdifferent wavelengths, or colors (e.g., different component colors, asdiscussed herein).

In some embodiments, the light injected into the waveguides 270, 280,290, 300, 310 is provided by a light projector system 520, whichcomprises a light module 530, which may include a light emitter, such asa light emitting diode (LED). The light from the light module 530 may bedirected to and modified by a light modulator 540, e.g., a spatial lightmodulator, via a beam splitter 550. The light modulator 540 may beconfigured to change the perceived intensity of the light injected intothe waveguides 270, 280, 290, 300, 310 to encode the light with imageinformation. Examples of spatial light modulators include liquid crystaldisplays (LCD) including a liquid crystal on silicon (LCOS) displays. Itwill be appreciated that the image injection devices 360, 370, 380, 390,400 are illustrated schematically and, in some embodiments, these imageinjection devices may represent different light paths and locations in acommon projection system configured to output light into associated onesof the waveguides 270, 280, 290, 300, 310. In some embodiments, thewaveguides of the waveguide assembly 260 may function as ideal lenswhile relaying light injected into the waveguides out to the user'seyes. In this conception, the object may be the spatial light modulator540 and the image may be the image on the depth plane.

In some embodiments, the display system 250 may be a scanning fiberdisplay comprising one or more scanning fibers configured to projectlight in various patterns (e.g., raster scan, spiral scan, Lissajouspatterns, etc.) into one or more waveguides 270, 280, 290, 300, 310 andultimately to the eye 210 of the viewer. In some embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a single scanning fiber or a bundle of scanningfibers configured to inject light into one or a plurality of thewaveguides 270, 280, 290, 300, 310. In some other embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a plurality of scanning fibers or a plurality ofbundles of scanning fibers, each of which are configured to inject lightinto an associated one of the waveguides 270, 280, 290, 300, 310. Itwill be appreciated that one or more optical fibers may be configured totransmit light from the light module 530 to the one or more waveguides270, 280, 290, 300, 310. It will be appreciated that one or moreintervening optical structures may be provided between the scanningfiber, or fibers, and the one or more waveguides 270, 280, 290, 300, 310to, e.g., redirect light exiting the scanning fiber into the one or morewaveguides 270, 280, 290, 300, 310.

A controller 560 controls the operation of one or more of the stackedwaveguide assembly 260, including operation of the image injectiondevices 360, 370, 380, 390, 400, the light source 530, and the lightmodulator 540. In some embodiments, the controller 560 is part of thelocal data processing module 140. The controller 560 includesprogramming (e.g., instructions in a non-transitory medium) thatregulates the timing and provision of image information to thewaveguides 270, 280, 290, 300, 310 according to, e.g., any of thevarious schemes disclosed herein. In some embodiments, the controllermay be a single integral device, or a distributed system connected bywired or wireless communication channels. The controller 560 may be partof the processing modules 70 or 72 (FIG. 2A) in some embodiments.

With continued reference to FIG. 1.10, the waveguides 270, 280, 290,300, 310 may be configured to propagate light within each respectivewaveguide by total internal reflection (TIR). The waveguides 270, 280,290, 300, 310 may each be planar or have another shape (e.g., curved),with major top and bottom surfaces and edges extending between thosemajor top and bottom surfaces. In the illustrated configuration, thewaveguides 270, 280, 290, 300, 310 may each include out-coupling opticalelements 570, 580, 590, 600, 610 that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 210. Extracted light may also be referred to as out-coupledlight and the out-coupling optical elements light may also be referredto light extracting optical elements. An extracted beam of light may beoutputted by the waveguide at locations at which the light propagatingin the waveguide strikes a light extracting optical element. Theout-coupling optical elements 570, 580, 590, 600, 610 may, for example,be gratings, including diffractive optical features, as discussedfurther herein. While illustrated disposed at the bottom major surfacesof the waveguides 270, 280, 290, 300, 310, for ease of description anddrawing clarity, in some embodiments, the out-coupling optical elements570, 580, 590, 600, 610 may be disposed at the top and/or bottom majorsurfaces, and/or may be disposed directly in the volume of thewaveguides 270, 280, 290, 300, 310, as discussed further herein. In someembodiments, the out-coupling optical elements 570, 580, 590, 600, 610may be formed in a layer of material that is attached to a transparentsubstrate to form the waveguides 270, 280, 290, 300, 310. In some otherembodiments, the waveguides 270, 280, 290, 300, 310 may be a monolithicpiece of material and the out-coupling optical elements 570, 580, 590,600, 610 may be formed on a surface and/or in the interior of that pieceof material.

With continued reference to FIG. 1.10, as discussed herein, eachwaveguide 270, 280, 290, 300, 310 is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 270 nearest the eye may be configured to deliver collimatedlight (which was injected into such waveguide 270), to the eye 210. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 280 may be configured to send outcollimated light which passes through the first lens 350 (e.g., anegative lens) before it may reach the eye 210; such first lens 350 maybe configured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 280 ascoming from a first focal plane closer inward toward the eye 210 fromoptical infinity. Similarly, the third up waveguide 290 passes itsoutput light through both the first 350 and second 340 lenses beforereaching the eye 210; the combined optical power of the first 350 andsecond 340 lenses may be configured to create another incremental amountof wavefront curvature so that the eye/brain interprets light comingfrom the third waveguide 290 as coming from a second focal plane that iseven closer inward toward the person from optical infinity than waslight from the next waveguide up 280.

The other waveguide layers 300, 310 and lenses 330, 320 are similarlyconfigured, with the highest waveguide 310 in the stack sending itsoutput through all of the lenses between it and the eye for an aggregatefocal power representative of the closest focal plane to the person. Tocompensate for the stack of lenses 320, 330, 340, 350 whenviewing/interpreting light coming from the world 510 on the other sideof the stacked waveguide assembly 260, a compensating lens layer 620 maybe disposed at the top of the stack to compensate for the aggregatepower of the lens stack 320, 330, 340, 350 below. Such a configurationprovides as many perceived focal planes as there are availablewaveguide/lens pairings. Both the out-coupling optical elements of thewaveguides and the focusing aspects of the lenses may be static (i.e.,not dynamic or electro-active). In some alternative embodiments, eitheror both may be dynamic using electro-active features.

In some embodiments, two or more of the waveguides 270, 280, 290, 300,310 may have the same associated depth plane. For example, multiplewaveguides 270, 280, 290, 300, 310 may be configured to output imagesset to the same depth plane, or multiple subsets of the waveguides 270,280, 290, 300, 310 may be configured to output images set to the sameplurality of depth planes, with one set for each depth plane. This mayprovide advantages for forming a tiled image to provide an expandedfield of view at those depth planes.

With continued reference to FIG. 1.10, the out-coupling optical elements570, 580, 590, 600, 610 may be configured to both redirect light out oftheir respective waveguides and to output this light with theappropriate amount of divergence or collimation for a particular depthplane associated with the waveguide. As a result, waveguides havingdifferent associated depth planes may have different configurations ofout-coupling optical elements 570, 580, 590, 600, 610, which outputlight with a different amount of divergence depending on the associateddepth plane. In some embodiments, the light extracting optical elements570, 580, 590, 600, 610 may be volumetric or surface features, which maybe configured to output light at specific angles. For example, the lightextracting optical elements 570, 580, 590, 600, 610 may be volumeholograms, surface holograms, and/or diffraction gratings. In someembodiments, the features 320, 330, 340, 350 may not be lenses; rather,they may simply be spacers (e.g., cladding layers and/or structures forforming air gaps).

In some embodiments, the out-coupling optical elements 570, 580, 590,600, 610 are diffractive features that form a diffraction pattern, or“diffractive optical element” (also referred to herein as a “DOE”).Preferably, the DOE's have a sufficiently low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye 210 with each intersection of the DOE, while the rest continuesto move through a waveguide via TIR. The light carrying the imageinformation is thus divided into a number of related exit beams thatexit the waveguide at a multiplicity of locations and the result is afairly uniform pattern of exit emission toward the eye 210 for thisparticular collimated beam bouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets may be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet may be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, a camera assembly 630 (e.g., a digital camera,including visible light and infrared light cameras) may be provided tocapture images of the eye 210 and/or tissue around the eye 210 to, e.g.,detect user inputs and/or to monitor the physiological state of theuser. As used herein, a camera may be any image capture device. In someembodiments, the camera assembly 630 may include an image capture deviceand a light source to project light (e.g., infrared light) to the eye,which may then be reflected by the eye and detected by the image capturedevice. In some embodiments, the camera assembly 630 may be attached tothe frame 54 (FIG. 2A) and may be in electrical communication with theprocessing modules 140 and/or 150, which may process image informationfrom the camera assembly 630. In some embodiments, one camera assembly630 may be utilized for each eye, to separately monitor each eye.

With reference now to FIG. 1.11, an example of exit beams outputted by awaveguide is shown. One waveguide is illustrated, but it will beappreciated that other waveguides in the waveguide assembly 260 (FIG.1.10) may function similarly, where the waveguide assembly 260 includesmultiple waveguides. Light 640 is injected into the waveguide 270 at theinput surface 460 of the waveguide 270 and propagates within thewaveguide 270 by TIR. At points where the light 640 impinges on the DOE570, a portion of the light exits the waveguide as exit beams 650. Theexit beams 650 are illustrated as substantially parallel but, asdiscussed herein, they may also be redirected to propagate to the eye210 at an angle (e.g., forming divergent exit beams), depending on thedepth plane associated with the waveguide 270. It will be appreciatedthat substantially parallel exit beams may be indicative of a waveguidewith out-coupling optical elements that out-couple light to form imagesthat appear to be set on a depth plane at a large distance (e.g.,optical infinity) from the eye 210. Other waveguides or other sets ofout-coupling optical elements may output an exit beam pattern that ismore divergent, which would require the eye 210 to accommodate to acloser distance to bring it into focus on the retina and would beinterpreted by the brain as light from a distance closer to the eye 210than optical infinity.

In some embodiments, a full color image may be formed at each depthplane by overlaying images in each of the component colors, e.g., threeor more component colors. FIG. 1.12 illustrates an example of a stackedwaveguide assembly in which each depth plane includes images formedusing multiple different component colors. The illustrated embodimentshows depth planes 240 a-240 f, although more or fewer depths are alsocontemplated. Each depth plane may have three or more component colorimages associated with it, including: a first image of a first color, G;a second image of a second color, R; and a third image of a third color,B. Different depth planes are indicated in the figure by differentnumbers for diopters (dpt) following the letters G, R, and B. Just asexamples, the numbers following each of these letters indicate diopters(l/m), or inverse distance of the depth plane from a viewer, and eachbox in the figures represents an individual component color image. Insome embodiments, to account for differences in the eye's focusing oflight of different wavelengths, the exact placement of the depth planesfor different component colors may vary. For example, differentcomponent color images for a given depth plane may be placed on depthplanes corresponding to different distances from the user. Such anarrangement may increase visual acuity and user comfort and/or maydecrease chromatic aberrations.

In some embodiments, light of each component color may be outputted by asingle dedicated waveguide and, consequently, each depth plane may havemultiple waveguides associated with it. In such embodiments, each box inthe figures including the letters G, R, or B may be understood torepresent an individual waveguide, and three waveguides may be providedper depth plane where three component color images are provided perdepth plane. While the waveguides associated with each depth plane areshown adjacent to one another in this drawing for ease of description,it will be appreciated that, in a physical device, the waveguides mayall be arranged in a stack with one waveguide per level. In some otherembodiments, multiple component colors may be outputted by the samewaveguide, such that, e.g., only a single waveguide may be provided perdepth plane.

With continued reference to FIG. 1.12, in some embodiments, G is thecolor green, R is the color red, and B is the color blue. In some otherembodiments, other colors associated with other wavelengths of light,including magenta and cyan, may be used in addition to or may replaceone or more of red, green, or blue.

It will be appreciated that references to a given color of lightthroughout this disclosure will be understood to encompass light of oneor more wavelengths within a range of wavelengths of light that areperceived by a viewer as being of that given color. For example, redlight may include light of one or more wavelengths in the range of about620-780 nm, green light may include light of one or more wavelengths inthe range of about 492-577 nm, and blue light may include light of oneor more wavelengths in the range of about 435-493 nm.

In some embodiments, the light source 530 (FIG. 1.10) may be configuredto emit light of one or more wavelengths outside the visual perceptionrange of the viewer, for example, infrared and/or ultravioletwavelengths. In addition, the in-coupling, out-coupling, and other lightredirecting structures of the waveguides of the display 250 may beconfigured to direct and emit this light out of the display towards theuser's eye 210, e.g., for imaging and/or user stimulation applications.

With reference now to FIG. 1.13, in some embodiments, light impinging ona waveguide may need to be redirected to in-couple that light into thewaveguide. An in-coupling optical element may be used to redirect andin-couple the light into its corresponding waveguide. FIG. 1.13illustrates a cross-sectional side view of an example of a plurality orset 660 of stacked waveguides that each includes an in-coupling opticalelement. The waveguides may each be configured to output light of one ormore different wavelengths, or one or more different ranges ofwavelengths. It will be appreciated that the stack 660 may correspond tothe stack 260 (FIG. 1.10) and the illustrated waveguides of the stack660 may correspond to part of the plurality of waveguides 270, 280, 290,300, 310, except that light from one or more of the image injectiondevices 360, 370, 380, 390, 400 is injected into the waveguides from aposition that requires light to be redirected for in-coupling.

The illustrated set 660 of stacked waveguides includes waveguides 670,680, and 690. Each waveguide includes an associated in-coupling opticalelement (which may also be referred to as a light input area on thewaveguide), with, e.g., in-coupling optical element 700 disposed on amajor surface (e.g., an upper major surface) of waveguide 670,in-coupling optical element 710 disposed on a major surface (e.g., anupper major surface) of waveguide 680, and in-coupling optical element720 disposed on a major surface (e.g., an upper major surface) ofwaveguide 690. In some embodiments, one or more of the in-couplingoptical elements 700, 710, 720 may be disposed on the bottom majorsurface of the respective waveguide 670, 680, 690 (particularly wherethe one or more in-coupling optical elements are reflective, deflectingoptical elements). As illustrated, the in-coupling optical elements 700,710, 720 may be disposed on the upper major surface of their respectivewaveguide 670, 680, 690 (or the top of the next lower waveguide),particularly where those in-coupling optical elements are transmissive,deflecting optical elements. In some embodiments, the in-couplingoptical elements 700, 710, 720 may be disposed in the body of therespective waveguide 670, 680, 690. In some embodiments, as discussedherein, the in-coupling optical elements 700, 710, 720 are wavelengthselective, such that they selectively redirect one or more wavelengthsof light, while transmitting other wavelengths of light. Whileillustrated on one side or corner of their respective waveguide 670,680, 690, it will be appreciated that the in-coupling optical elements700, 710, 720 may be disposed in other areas of their respectivewaveguide 670, 680, 690 in some embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 1.10, and may be separated (e.g.,laterally spaced apart) from other in-coupling optical elements 700,710, 720 such that it substantially does not receive light from theother ones of the in-coupling optical elements 700, 710, 720.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750, may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750, may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690, respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, and/or solid layers of material. For example, asillustrated, layer 760 a may separate waveguides 670 and 680; and layer760 b may separate waveguides 680 and 690. In some embodiments, thelayers 760 a and 760 b are formed of low refractive index materials(that is, materials having a lower refractive index than the materialforming the immediately adjacent one of waveguides 670, 680, 690).Preferably, the refractive index of the material forming the layers 760a, 760 b is 0.05 or more, or 0.10 or less than the refractive index ofthe material forming the waveguides 670, 680, 690. Advantageously, thelower refractive index layers 760 a, 760 b may function as claddinglayers that facilitate total internal reflection (TIR) of light throughthe waveguides 670, 680, 690 (e.g., TIR between the top and bottom majorsurfaces of each waveguide). In some embodiments, the layers 760 a, 760b are formed of air. While not illustrated, it will be appreciated thatthe top and bottom of the illustrated set 660 of waveguides may includeimmediately neighboring cladding layers.

Preferably, for ease of manufacturing and other considerations, thematerial forming the waveguides 670, 680, 690 are similar or the same,and the material forming the layers 760 a, 760 b are similar or thesame. In some embodiments, the material forming the waveguides 670, 680,690 may be different between one or more waveguides, and/or the materialforming the layers 760 a, 760 b may be different, while still holding tothe various refractive index relationships noted above.

With continued reference to FIG. 1.13, light rays 770, 780, 790 areincident on the set 660 of waveguides. It will be appreciated that thelight rays 770, 780, 790 may be injected into the waveguides 670, 680,690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG.1.10).

In some embodiments, the light rays 770, 780, 790 have differentproperties, e.g., different wavelengths or different ranges ofwavelengths, which may correspond to different colors. The in-couplingoptical elements 700, 710, 720 each deflect the incident light such thatthe light propagates through a respective one of the waveguides 670,680, 690 by TIR. In some embodiments, the incoupling optical elements700, 710, 720 each selectively deflect one or more particularwavelengths of light, while transmitting other wavelengths to anunderlying waveguide and associated incoupling optical element.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths,while transmitting rays 780 and 790, which have different second andthird wavelengths or ranges of wavelengths, respectively. Thetransmitted ray 780 impinges on and is deflected by the in-couplingoptical element 710, which is configured to deflect light of a secondwavelength or range of wavelengths. The ray 790 is deflected by thein-coupling optical element 720, which is configured to selectivelydeflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 1.13, the deflected light rays 770,780, 790 are deflected so that they propagate through a correspondingwaveguide 670, 680, 690; that is, the in-coupling optical elements 700,710, 720 of each waveguide deflects light into that correspondingwaveguide 670, 680, 690 to in-couple light into that correspondingwaveguide. The light rays 770, 780, 790 are deflected at angles thatcause the light to propagate through the respective waveguide 670, 680,690 by TIR. The light rays 770, 780, 790 propagate through therespective waveguide 670, 680, 690 by TIR until impinging on thewaveguide's corresponding light distributing elements 730, 740, 750.

With reference now to FIG. 1.14, a perspective view of an example of theplurality of stacked waveguides of FIG. 1.13 is illustrated. As notedabove, the incoupled light rays 770, 780, 790, are deflected by thein-coupling optical elements 700, 710, 720, respectively, and thenpropagate by TIR within the waveguides 670, 680, 690, respectively. Thelight rays 770, 780, 790 then impinge on the light distributing elements730, 740, 750, respectively. The light distributing elements 730, 740,750 deflect the light rays 770, 780, 790 so that they propagate towardsthe out-coupling optical elements 800, 810, 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE's). In some embodiments, the OPE'sdeflect or distribute light to the out-coupling optical elements 800,810, 820 and, in some embodiments, may also increase the beam or spotsize of this light as it propagates to the out-coupling opticalelements. In some embodiments, the light distributing elements 730, 740,750 may be omitted and the in-coupling optical elements 700, 710, 720may be configured to deflect light directly to the out-coupling opticalelements 800, 810, 820. For example, with reference to FIG. 1.13, thelight distributing elements 730, 740, 750 may be replaced without-coupling optical elements 800, 810, 820, respectively. In someembodiments, the out-coupling optical elements 800, 810, 820 are exitpupils (EP's) or exit pupil expanders (EPE's) that direct light in aviewer's eye 210 (FIG. 1.11). It will be appreciated that the OPE's maybe configured to increase the dimensions of the eye box in at least oneaxis and the EPE's may be configured to increase the eye box in an axiscrossing, e.g., orthogonal to, the axis of the OPEs. For example, eachOPE may be configured to redirect a portion of the light striking theOPE to an EPE of the same waveguide, while allowing the remainingportion of the light to continue to propagate down the waveguide. Uponimpinging on the OPE again, another portion of the remaining light isredirected to the EPE, and the remaining portion of that portioncontinues to propagate further down the waveguide, and so on. Similarly,upon striking the EPE, a portion of the impinging light is directed outof the waveguide towards the user, and a remaining portion of that lightcontinues to propagate through the waveguide until it strikes the EPagain, at which time another portion of the impinging light is directedout of the waveguide, and so on. Consequently, a single beam ofincoupled light may be “replicated” each time a portion of that light isredirected by an OPE or EPE, thereby forming a field of cloned beams oflight, as shown in FIG. 1.10. In some embodiments, the OPE and/or EPEmay be configured to modify a size of the beams of light.

Accordingly, with reference to FIGS. 1.13 and 1.14, in some embodiments,the set 660 of waveguides includes waveguides 670, 680, 690; in-couplingoptical elements 700, 710, 720; light distributing elements (e.g.,OPE's) 730, 740, 750; and out-coupling optical elements (e.g., EP's)800, 810, 820 for each component color. The waveguides 670, 680, 690 maybe stacked with an air gap/cladding layer between each one. Thein-coupling optical elements 700, 710, 720 redirect or deflect incidentlight (with different in-coupling optical elements receiving light ofdifferent wavelengths) into its waveguide. The light then propagates atan angle which will result in TIR within the respective waveguide 670,680, 690. In the example shown, light ray 770 (e.g., blue light) isdeflected by the first in-coupling optical element 700, and thencontinues to bounce down the waveguide, interacting with the lightdistributing element (e.g., OPE's) 730 and then the out-coupling opticalelement (e.g., EPs) 800, in a manner described earlier. The light rays780 and 790 (e.g., green and red light, respectively) will pass throughthe waveguide 670, with light ray 780 impinging on and being deflectedby in-coupling optical element 710. The light ray 780 then bounces downthe waveguide 680 via TIR, proceeding on to its light distributingelement (e.g., OPEs) 740 and then the out-coupling optical element(e.g., EP's) 810. Finally, light ray 790 (e.g., red light) passesthrough the waveguide 690 to impinge on the light in-coupling opticalelements 720 of the waveguide 690. The light in-coupling opticalelements 720 deflect the light ray 790 such that the light raypropagates to light distributing element (e.g., OPEs) 750 by TIR, andthen to the out-coupling optical element (e.g., EPs) 820 by TIR. Theout-coupling optical element 820 then finally out-couples the light ray790 to the viewer, who also receives the out-coupled light from theother waveguides 670, 680.

FIG. 1.15 illustrates a top-down plan view of an example of theplurality of stacked waveguides of FIGS. 1.13 and 1.14. As illustrated,the waveguides 670, 680, 690, along with each waveguide's associatedlight distributing element 730, 740, 750 and associated out-couplingoptical element 800, 810, 820, may be vertically aligned. However, asdiscussed herein, the in-coupling optical elements 700, 710, 720 are notvertically aligned; rather, the in-coupling optical elements arepreferably non-overlapping (e.g., laterally spaced apart as seen in thetop-down view). As discussed further herein, this nonoverlapping spatialarrangement facilitates the injection of light from different resourcesinto different waveguides on a one-to-one basis, thereby allowing aspecific light source to be uniquely coupled to a specific waveguide. Insome embodiments, arrangements including nonoverlappingspatially-separated in-coupling optical elements may be referred to as ashifted pupil system, and the in-coupling optical elements within thesearrangements may correspond to sub pupils.

Referring to FIGS. 2A-2D, some general componentry options areillustrated. In the portions of the detailed description which followthe discussion of FIGS. 2A-2D, various systems, subsystems, andcomponents are presented for addressing the objectives of providing ahigh-quality, comfortably-perceived display system for human VR and/orAR that access and create external information sources.

As shown in FIG. 2A, an AR system user (60) is depicted wearing headmounted component (58) featuring a frame (64) structure coupled to adisplay system (62) positioned in front of the eyes of the user. Aspeaker (66) is coupled to the frame (64) in the depicted configurationand positioned adjacent the ear canal of the user (in one embodiment,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display(62) is operatively coupled (68), such as by a wired lead or wirelessconnectivity, to a local processing and data module (70) which may bemounted in a variety of configurations, such as fixedly attached to theframe (64), fixedly attached to a helmet or hat (80) as shown in theembodiment of FIG. 2B, embedded in headphones, removably attached to thetorso (82) of the user (60) in a backpack-style configuration as shownin the embodiment of FIG. 2C, or removably attached to the hip (84) ofthe user (60) in a belt-coupling style configuration as shown in theembodiment of FIG. 2D.

The local processing and data module (70) may comprise a processor orcontroller (e.g., a power-efficient processor or controller), as well asdigital memory, such as flash memory, both of which may be utilized toassist in the processing, caching, and storage of data a) captured fromsensors which may be operatively coupled to the frame (64), such aselectromagnetic emitters and detectors, image capture devices (such ascameras), microphones, inertial measurement units, accelerometers,compasses, GPS units, radio devices, and/or gyros; and/or b) acquiredand/or processed using the remote processing module (72) and/or remotedata repository (74), possibly for passage to the display (62) aftersuch processing or retrieval. The local processing and data module (70)may be operatively coupled (76, 78), such as via a wired or wirelesscommunication links, to the remote processing module (72) and remotedata repository (74) such that these remote modules (72, 74) areoperatively coupled to each other and available as resources to thelocal processing and data module (70).

In one embodiment, the remote processing module (72) may comprise one ormore relatively powerful processors or controllers configured to analyzeand process data, light properties emitted or received, and/or imageinformation. In one embodiment, the remote data repository (74) maycomprise a relatively large-scale digital data storage facility, whichmay be available through the internet or other networking configurationin a “cloud” resource configuration. In one embodiment, all data isstored and all computation is performed in the local processing and datamodule, allowing fully autonomous use from any remote modules.

Referring now to FIG. 3, a schematic illustrates coordination betweenthe cloud computing assets (46) and local processing assets, which may,for example reside in head mounted components (58) coupled to the user'shead (120) and a local processing and data module (70), coupled to theuser's belt (308); therefore the component 70 may also be termed a “beltpack” 70), as shown in FIG. 3. In one embodiment, the cloud (46) assets,such as one or more server systems (110) are operatively coupled (115),such as via wired or wireless networking (wireless generally beingpreferred for mobility, wired generally being preferred for certainhigh-bandwidth or high-data-volume transfers that may be desired),directly to (40, 42) one or both of the local computing assets, such asprocessor and memory configurations, coupled to the user's head (120)and belt (308) as described above. These computing assets local to theuser may be operatively coupled to each other as well, via wired and/orwireless connectivity configurations (44), such as the wired coupling(68) discussed below in reference to FIG. 8.

In one embodiment, to maintain a low-inertia and small-size subsystemmounted to the user's head (120), primary transfer between the user andthe cloud (46) may be via the link between the subsystem mounted at thebelt (308) and the cloud, with the head mounted (120) subsystemprimarily data-tethered to the belt-based (308) subsystem using wirelessconnectivity, such as ultra-wideband (“UWB”) connectivity, as iscurrently employed, for example, in personal computing peripheralconnectivity applications.

With efficient local and remote processing coordination, and anappropriate display device for a user, such as the user interface oruser display system (62) shown in FIG. 2A, or variations thereof,aspects of one world pertinent to a user's current actual or virtuallocation may be transferred or “passed” to the user and updated in anefficient fashion. In other words, a map of the world may be continuallyupdated at a storage location which may, e.g., partially reside on theuser's AR system and partially reside in the cloud resources. The map(also referred to as a “passable world model”) may be a large databasecomprising raster imagery, 3-D and 2-D points, parametric informationand other information about the real world. As more and more AR userscontinually capture information about their real environment (e.g.,through cameras, sensors, IMUs, etc.), the map becomes more and moreaccurate and complete.

With a configuration as described above, wherein there is one worldmodel that can reside on cloud computing resources and be distributedfrom there, such world can be “passable” to one or more users in arelatively low bandwidth form preferable to trying to pass aroundreal-time video data or the like. In some embodiments, the augmentedexperience of the person standing near the statue (i.e., as shown inFIG. 1) may be informed by the cloud-based world model, a subset ofwhich may be passed down to them and their local display device tocomplete the view. A person sitting at a remote display device, whichmay be as simple as a personal computer sitting on a desk, canefficiently download that same section of information from the cloud andhave it rendered on their display. Indeed, one person actually presentin the park near the statue may take a remotely-located friend for awalk in that park, with the friend joining through virtual and augmentedreality. The system will need to know where the street is, where thetrees are, where the statue is—but with that information on the cloud,the joining friend can download from the cloud aspects of the scenario,and then start walking along as an augmented reality local relative tothe person who is actually in the park.

3-D points may be captured from the environment, and the pose (i.e.,vector and/or origin position information relative to the world) of thecameras that capture those images or points may be determined, so thatthese points or images may be “tagged”, or associated, with this poseinformation. Then points captured by a second camera may be utilized todetermine the pose of the second camera. In other words, one can orientand/or localize a second camera based upon comparisons with taggedimages from a first camera. Then this knowledge may be utilized toextract textures, make maps, and create a virtual copy of the real world(because then there are two cameras around that are registered).

So, at the base level, in some embodiments a person-worn system may beutilized to capture both 3-D points and the 2-D images that produced thepoints, and these points and images may be sent out to a cloud storageand processing resource. They may also be cached locally with embeddedpose information (e.g., cache the tagged images); so, the cloud may haveon the ready (e.g., in available cache) tagged 2-D images (e.g., taggedwith a 3-D pose), along with 3-D points. If a user is observingsomething dynamic (e.g., a scene with moving objects or features),he/she may also send additional information up to the cloud pertinent tothe motion (for example, if looking at another person's face, the usercan take a texture map of the face and push that up at an optimizedfrequency even though the surrounding world is otherwise basicallystatic). As noted above, more information on object recognizers and thepassable world model may be found in U.S. patent application Ser. No.14/205,126, entitled “System and method for augmented and virtualreality”, which is incorporated by reference in its entirety herein,along with the following additional disclosures, which relate toaugmented and virtual reality systems such as those developed by MagicLeap, Inc. of Fort Lauderdale, Fla.: U.S. patent application Ser. No.14/641,376; U.S. patent application Ser. No. 14/555,585; U.S. patentapplication Ser. No. 14/212,961; U.S. patent application Ser. No.14/690,401; U.S. patent application Ser. No. 13/663,466; U.S. patentapplication Ser. No. 13/684,489; and U.S. Patent Application Ser. No.62/298,993, each of which is incorporated by reference herein in itsentirety.

In some embodiments, the use of such passable world information maypermit identification and labelling of objects by spectroscopy to thenpass between users. For example, in a clinical setting, a firstcaregiver operating a device implementing features of the presentdisclosure may map and detect cancerous tissue on a patient and assignand apply a virtual label, much like a metatag, to the tissue. A secondcaregiver similarly wearing such a device may then look at the samecancerous tissue cell cluster and receive notice of the virtual labelidentifying such cells without needing to engage in one or more ofemitting light, receiving light, matching an absorption trait to atissue, and labeling the tissue independently.

GPS and other localization information may be utilized as inputs to suchprocessing. It will be appreciated that highly accurate localization ofthe user's head, totems, hand gestures, haptic devices etc. canfacilitate displaying appropriate virtual content to the user, orpassable virtual or augmented content among users in a passable world.

Referring to FIG. 5, a top orthogonal view of a head mountable component(58) of a wearable computing configuration is illustrated featuringvarious integrated components for an exemplary spectroscopy system. Theconfiguration features two display elements (62—binocular—one for eacheye), two forward-oriented cameras (124) for observing and detecting theworld around the user, each camera (124) having an associated field ofview (18, 22), and at least one spectroscopy array (126, described ingreater detail in FIG. 6), with a field of view (20); also aforward-oriented relatively high resolution picture camera (156) with afield of view (26), one or more inertial measurement units (102), and adepth sensor (154) with an associated field of view (24), such asdescribed in the aforementioned incorporated by reference disclosures.Facing toward the eyes (12, 13) of the user and coupled to the headmounted component (58) frame are eye tracking cameras (828, 830) andinward emitters and receivers (832, 834). One of skill in the art willappreciate the inward emitters and receivers (832, 834) emit and receivelight directed towards the eyes in irradiation pattern (824, 826) muchin the same way spectroscopy array (126) does for outward objects in itsfield of view (20). These components, or combinations less inclusive ofall components are operatively coupled such as by wire lead, to acontroller (844), which is operatively coupled (848) to a power supply(846), such as a battery.

In some embodiments, the head mountable component (58) may furtherinclude an ambient light detector (128) and/or an anti-scatter grid(129). The ambient light detector (128) includes at least onephotodetector and may be oriented outward (e.g., generallyforward-oriented) so as to capture ambient light from the world, or theambient environment around the user. In some embodiments, the ambientlight detector (128) may be forward-oriented, similar to the forwardoriented cameras (124), such that ambient light may be detected whilethe spectroscopy array (126) is not emitting light for spectroscopicanalysis. In another example, the ambient light detector (128) may beoriented outward in a non-forward direction (e.g., left, right, up, ordown) such that ambient light may be detected independent of whether thespectroscopy array (126) is emitting light. The anti-scatter grid (129)may be located such that reflected light travels therethrough beforebeing detected at the spectroscopy array (126). Ambient light detector(128) and anti-scatter grid (129) will be discussed in greater detailwith reference to FIG. 6.

In some embodiments, the display elements (62) include one or morewaveguides (e.g., a waveguide stack) which are optically transmissiveand allow the user to “see” the world by receiving light from the world.The waveguides also receive light containing display information andpropagate and eject the light to the user's eyes (12, 13), to therebydisplay an image to the user. Preferably, light propagating out of thewaveguide provides particular, defined levels of wavefront divergencecorresponding to different depth planes (e.g., the light forming animage of an object at a particular distance from the user has awavefront divergence that corresponds to or substantially matches thewavefront divergence of light that would reach the user from that objectif real). For example, the waveguides may have optical power and may beconfigured to output light with selectively variable levels of wavefrontdivergence. It will be appreciated that this wavefront divergenceprovides cues to accommodation for the eyes (12, 13). In addition, thedisplay elements (62) utilize binocular disparity to further providedepth cues, e.g. cues to vergence of the eyes (12, 13). Advantageously,the cues to accommodation and cues to vergence may match, e.g., suchthat they both correspond to an object at the same distance from theuser. This accommodation-vergence matching facilitates the long-termwearability of a system utilizing the head-mounted member (58).

With continued reference to FIG. 5, preferably, each emitter (126, 832,834) is configured to controllably emit electromagnetic radiation in twoor more wavelengths, such as about 660 nm, and about 940 nm, such as byLEDs, and preferably the fields of irradiation (824, 826) are orientedto irradiate targeted objects or surfaces. In some embodiments, targetedobjects are inward, such as eyes (12, 13) and irradiation patterns (824,826) may be fixed or broadened/narrowed to target specific areas of aneye in response to an eye tracking camera data point. In someembodiments, targeted objects are outward (e.g., away from the user),and the irradiation pattern within the field of view (20) ofspectroscope array (126) conforms to a gaze of the eyes (12, 13)determined from eye tracking cameras (828, 830).

In some embodiments, the gaze may be understood to be a vector extendingfrom the user's eye, such as extending from the fovea through the lensof the eye, and the emitters (832, 834) may output infrared light on theuser's eyes, and reflections from the eye (e.g., corneal reflections)may be monitored. A vector between a pupil center of an eye (e.g., thedisplay system may determine a centroid of the pupil, for instancethrough infrared imaging) and the reflections from the eye may be usedto determine the gaze of the eye. In some embodiments, when estimatingthe position of the eye, since the eye has a sclera and an eyeball, thegeometry can be represented as two circles layered on top of each other.The eye pointing vector may be determined or calculated based on thisinformation. Also the eye center of rotation may be estimated since thecross section of the eye is circular and the sclera swings through aparticular angle. This may result in a vector distance because ofautocorrelation of the received signal against known transmitted signal,not just ray traces. The output may be seen as a Purkinje image 1400which may in turn be used to track movement of the eyes.

One of skill in the art will appreciate other ways to determine anirradiation pattern within field of view (20) such as by head poseinformation determined by one or more of IMU (102).

In some embodiments, the emitters may be configured to emit wavelengthssimultaneously, or sequentially, with controlled pulsatile emissioncycling. The one or more detectors (126, 828, 830) may comprisephotodiodes, photodetectors, and/or digital camera sensors (e.g., CCD orCMOS image sensors), and preferably are positioned and oriented toreceive radiation that has encountered the targeted tissue or materialor object otherwise. The one or more electromagnetic radiation detectors(126, 828, 830) may comprise a digital image sensor comprising aplurality of pixels, wherein the controller (844) is configured toautomatically detect a subset of pixels which are receiving the lightreflected after encountering a target object, and to use such subset ofpixels to produce an output.

In some embodiments, the output is a function of matching received lightagainst emitted light to a target from an absorption database ofmaterials and material properties. For example, in some embodiments, anabsorption database comprises a plurality of absorption charts such asdepicted in FIGS. 7A and 7B. It will be appreciated that a databasecomprising charts may include electronic representations ortransformations of the information in the charts, and the use of theterm charts herein includes such representations or transformations.FIGS. 7A and 7B is merely used as an example, but demonstrates varioustissue properties that may be detected from a given system emittinglight from a particular light source and receiving light of a particularwavelength and/or light property to determine the probability of anobserved target being a particular tissue or having particularproperties within the tissue. Other charts, such as either saturationcurves or calibration curves, may be selectively accessed by a user. Forexample, a user could choose absorption databases for a particular lightsource or wavelength patterns and then look around until thespectroscopy system identifies material matching the propertiesrequested. Such an embodiment may be termed a “closed search,” or onethat looks for specific properties as opposed to an “open search” thatlooks at any target and then searches databases for matches on the lightproperties detected.

The controller (844) may be configured to automatically detect a subsetof pixels within a field of view (124, or 126, or 824, 826, FIG. 5)based at least in part upon reflected light properties differencesamongst signals associated with the pixels. For example, the controller(844) may be configured to automatically detect the subset of pixelsbased at least in part upon reflected light absorption differencesamongst signals associated with the pixels. Without being limited bytheory, light impacting upon an object will reflect, transmit (absorb),or scatter upon striking the object, such that R+T+S=1 (withR=reflection from the object, T=transmission/absorption into the object,and S=scatter from the object). If a particular subset of pixelsreflects a higher proportion of light relative to surrounding subpixels,the controller may isolate these subpixels or note or register the pixellocation for these different properties in a memory system. In someembodiments, the pixel location are stored in a passable world mappingsystem as dense or sparse mapping points such as additional users of ahead mounted display system access the map, the subset of pixels arepassed to the additional user and accessed and/or displayed on thesecond user's display.

Referring to FIG. 6, a spectroscopy array (126) may comprise a lightsource (612) emitting light (613) towards a target object (620). In someembodiments, the light source (612) is an electromagnetic emitter suchas light emitting diodes. In some embodiments, the direction of emittedlight (613) is substantially the same as a gaze orientation of a user(60) or a head pose orientation of a user (60). In some embodiments,photodetectors (614) capture reflected light (615) from the targetobject. In some embodiments, a processor (611), which may be controller(844) depicted in FIG. 5, determines an absorption property betweenemitted light (613) and reflected light (615) and matches the propertyfrom absorption database (630). In some embodiments, absorption database(630) is stored on a local processing module such as module (70)depicted in FIG. 2A for example; in some embodiments, absorptiondatabase (630) is stored on remote processing module (72) such as theone depicted in FIG. 2A.

In some embodiments, the spectroscopy array (126) further includes anambient light detector (628), which may correspond to the ambient lightdetector (128) of FIG. 5. The ambient light detector (628) may be anytype of electromagnetic radiation detector (e.g., a photodiode, aphotodetector, digital camera sensor, CCD, CMOS) configured to detectlight of the wavelengths produced by the light source (612) and/or arange of wavelengths similar to the wavelengths produced by the lightsource (612). The ambient light detector (628) may further be orientedsimilarly to the photodetectors (614) such that the direction and/orfield of view imaged by the ambient light detector (628) is the same orsimilar to the direction and/or field of view imaged by thephotodetectors (614). In some other embodiments, the ambient lightdetector (628) may be oriented to capture light from a differentdirection than the photodetectors (614). Such an arrangement may haveadvantages for guarding against unintentionally detecting reflectedlight (615).

The ambient light detector (628) may be configured to monitor ambientlight continuously or at discrete intervals. In addition, ambient lightdetector (628) may be configured to monitor ambient light during timeswhen the light source (612) is not emitting light for spectroscopicmeasurements, and/or during times when the light source (612) isemitting light but the system is not oriented toward a target object(620).

In some embodiments, the spectroscopy array (126) further includes ananti-scatter grid (629). In some embodiments, the anti-scatter grid(629) includes a grid of walls, defining openings therebetween. Light(615) travels through the grid in order to be received by thephotodetectors (614). Preferably, the walls extend substantiallyparallel to the direction of propagation of the light (615), therebydefining openings through which that light may propagate to impinge onthe photodetectors (614). In some embodiments, as seen in a view of theforward face of the anti-scatter grid (629), the openings may be in theshape of rectangles or squares. If someone environments, the openingsmay have any desired shape, e.g., circular, hexagonal, etc.

In some embodiments, the anti-scatter grid (629) make include aplurality of parallel components configured to attenuate light incidenton the anti-scatter grid (629) that is not propagating generallyperpendicular to the anti-scatter grid (629) (e.g., light that is withina range such as 1, 5°, 10°, etc., from perpendicular may be attenuated).The anti-scatter grid (629) is disposed along the path between thephotodetectors (614) and the world, such that light being capture by thephotodetectors preferably passes through the anti-scatter grid (629)before being captured or imaged. Thus, when performing spectroscopicmethods according to some embodiments, light from the light source (612)that is directed back to the photodetectors (614) by retroreflection atthe target object (620) may be generally perpendicular to theanti-scatter grid (629) and is not attenuated or minimally attenuated.However, scattered light and/or ambient light from other sources thatmay be present is likely to be propagating at larger angles relative toperpendicular. Thus, at least some ambient and scattered light isattenuated at the anti-scatter grid (629), thus reducing or eliminatingits contribution to the light measured at the photodetectors (614).

In some embodiments, the spectroscopy array (126) may further include ascatter photodetector (631) configured to capture light reflected fromthe anti-scatter grid (629). For example, some embodiments of theanti-scatter grid (629) may include walls with a thickness sufficient toprovide a surface, on the forward face of the anti-scatter grid (629),for scattered light to reflect off. The portion of the walls on theforward face of the anti-scatter grid (629) may be referred to as theforward face of the walls. In such embodiments, a fraction of the lightincident on the anti-scatter grid (629) may pass through the openingdefined between the walls of the anti-scatter grid (section 29) and theremainder of the light may be reflected by the forward face of thosewalls. Thus, the reflected light (615) incident on the forward face ofthe walls may be reflected away from the photodetectors (614), whilescattered or ambient light incident on the forward face of the walls mayreflect at an angle from the anti-scatter grid (629) and to be capturedby the scatter photodetector (631), which is oriented to receive thescattered light. In some environments, it may be assumed that thefraction of light reflected by the forward face of the walls of theanti-scatter grid (629) is approximately equal to the fraction of theoverall surface area of the anti-scatter grid (629) occupied by theforward face of the walls. Thus, the known fraction of the overallsurface area occupied by the forward face of the walls may further beused (e.g., at the processor (611)) to adjust the amount of lightmeasured at the scatter photodetector (631) to adjust for the reductionin intensity due to the presence of the anti-scatter grid (629). Thatis, part of the reflected light (615) may be blocked by the anti-scattergrid (629). Since the surface area of the forward face of the walls ofthe anti-scatter grid (629) may be known, and assuming that the amountof light blocked is proportional (e.g., roughly equal) to the surfacearea of that forward face of the walls, then the amount of lightreceived by the photodetectors (614) may be scaled up to account forlight that is blocked by the walls of the anti-scatter grid (629).

Object (620) is depicted as an apple in FIG. 6 for simplicity, andthough food properties have their respective light absorption propertiesand embodiments of the invention may be used to identify food by itslight properties, more sophisticated uses are also envisioned. In someembodiments, outward facing spectroscopy array (126) identifies tissuesource (624), e.g., an arm as depicted for illustrative purposes.Emitted light (613) may impact upon tissue source (624) and reflectedlight (615) may indicate the presence of irregular cells (626) amongstregular cells (625). As light source (612) irradiates tissue source(624), irregular cells (626) will return a different light property tophotodetectors (614) than regular cells (625). Irregular cells (626) maybe cancerous, be part of scar tissue, or even healthy cells amongst thetissue simply indicating or having a difference with surrounding cells,for example indicating where blood vessels or bone within tissue source(624) may be located. In some embodiments, regular cells constitute themajority of cells in a sample under analysis and irregular cellsconstitute a minority of the cells of the sample, the irregular cellsexhibiting a different detectable property than the regular cells. Insome embodiments, real world cameras capturing images on a pixel levelmay mark such irregular cells (626). As previously described, one suchmarking may be a labeling system applying a textual image proximate tothe irregular cells (626), another such labeling system may be a coloroverlay onto irregular cells (626), as seen through the display element62 (FIG. 5).

Thus, with reference again to FIG. 5, a system is presented fordetermining tissue properties or materials otherwise through a wearablecomputing system, such as one for AR or VR, comprising: a head-mountedmember (58) removably coupleable to the user's head; one or moreelectromagnetic radiation emitters (126, 832, 834) coupled to thehead-mounted member (58) and configured to emit light with at least twodifferent wavelengths in inward directions or outwards directions, oneor more electromagnetic radiation detectors (126, 828, 830) coupled tothe head-mounted member and configured to receive light reflected afterencountering a target object; and a controller (844) operatively coupledto the one or more electromagnetic radiation emitters (126, 832, 834)and one or more electromagnetic radiation detectors (126, 828, 830) andconfigured to cause the one or more electromagnetic radiation emittersto emit pulses of light while also causing the one or moreelectromagnetic radiation detectors to detect levels of light absorptionrelated to the emitted pulses of light, and to produce a displayableoutput.

The head-mounted member (58) may comprise frame configured to fit on theuser's head, e.g., an eyeglasses frame. The eyeglasses frame may be abinocular eyeglasses frame; alternative embodiments may be monocular.The one or more emitters (126, 832, and 834) may comprise a lightsource, for example at least one light emitting diode or otherelectromagnetic radiation emitter, emitting light at multiplewavelengths. The plurality of light sources may be configured topreferably emit at two wavelengths of light, e.g., a first wavelength ofabout 660 nanometers, and a second wavelength of about 940 nanometers.

In some embodiments, the one or more emitters (126, 832, 834) may beconfigured to emit light at the respective wavelengths sequentially. Insome embodiments, the one or more emitters (126, 832, 834) may beconfigured to emit light at the respective wavelengths simultaneously.The one or more electromagnetic radiation detectors (126, 828, 830) maycomprise a device selected from the group consisting of: a photodiode, aphotodetector, and a digital camera sensor. The controller (844) may befurther configured to cause the plurality of light emitting diodes toemit a cyclic pattern of first wavelength on, then second wavelength on,then both wavelengths off, such that the one or more electromagneticradiation detectors detect the first and second wavelengths separately.The controller (844) may be configured to cause the plurality of lightemitting diodes to emit a cyclic pattern of first wavelength on, thensecond wavelength on, then both wavelengths off, in a cyclic pulsingpattern about thirty times per second. The controller (844) may beconfigured to calculate a ratio of first wavelength light measurement tosecond wavelength light measurement, and wherein this ratio is convertedto an oxygen saturation reading via a lookup table based at least inpart upon the Beer-Lambert law.

The controller (844) may be configured to operate the one or moreemitters (126, 832, 834) and one or more electromagnetic radiationdetectors (126, 828, 830) to function as a head-mounted spectroscope.The controller (844) may be operatively coupled to an optical element(62) coupled to the head-mounted member (58) and viewable by the user,such that the output of the controller (844) that is indicative of aparticular characteristic of the target object, such as a materialproperty or tissue property of the target object, may be viewed by theuser through the optical element (62).

FIG. 7A is an example light property absorption chart that may bereferenced by an absorption database (630, FIG. 6). As depicted, variouslight source types, such as IR, NIR, or light emitting diodes in thevisible spectrum may be optimal for detecting certain tissues andproperties within the tissue. For example, pigmented lesions may havenoticeable absorption spectrum features between approximately 390 nm and1110 nm, while bulk tissue may have noticeable absorption spectrumfeatures between approximately 1155 nm and 1915 nm. In another example,blood vessels may produce detectable absorption peaks at betweenapproximately 390 nm and 455 nm, between approximately 525 nm and 610nm, and/or between approximately 770 nm and 1170 nm. In someembodiments, an absorption ratio or scatter in calibration curve iscomputed from emitted light to reflected light and applied to the givenabsorption database (630) such as depicted in FIG. 7A to determine theunderlying tissue and/or properties within or determine abnormalities.Moreover, light sources may be selected in order to selectivelyilluminate tissue with light in a desired wavelength range. For example,in some embodiments a gallium nitride light emitting diode (LED) mayemit light having a wavelength between approximately 390 nm and 455 nm,a gallium phosphide LED may emit light having a wavelength betweenapproximately 500 m and 690 nm, a gallium arsenide LED may emit lighthaving a wavelength between approximately 720 nm and 1090 nm, and indiumphosphide LED may emit light having a wavelength between approximately1110 nm and 1905 nm, and a gallium antimonide LED may emit light havinga wavelengths between approximately 1925 nm and over 2000 nm.

FIG. 7B depicts potential “overlap” of wavelengths. As depicted,“oxygenated blood” may overlap with “deoxygenated blood” at certainwavelengths, muting the results that a spectroscopic processes mayprovide. To avoid this potential overlap, in some embodiments, light ata second different wavelength is emitted to provide a second source oflight to measure and compare. For example, light may be emitted at afirst wavelength range between, e.g., 650 nm and 950 nm, and a secondwavelength range between, e.g., 1000 nm and 1350 nm.

FIG. 8 illustrates a method (850) for using a wearable AR/VR systemfeaturing spectroscopy components to identify a material or propertieswithin a material. Method (850) begins at (851) with the systemorienting light sources to a target object. In some embodiments, theorienting has light sources directed inwards towards the eyes of a user,and may be fixed or scanning the eye such as scanning the retina. Insome embodiments, the orienting is by determining an eye gaze or headpose of the user and orienting a light source in substantially the samedirection towards a target object within such gaze or pose field ofview, or towards feature landmarks or target objects.

In some embodiments, at (852) light sources emit light in an irradiationpattern towards the target object or surface. In some embodiments, thelight is pulsed at timed intervals by a timer. In some embodiments, thelight source emits light of at least one wavelength and at (854)radiation detectors, such as photo detectors, receive reflected light.In some embodiments, the detectors are also operatively coupled to atimer to indicate if received light was initially pulsed at a certaintime to determine changes in light properties upon reflecting on thetarget object. In some embodiments, (852) begins concurrent with mappingat (853) but this sequence is not necessarily so.

In some embodiments, real world capturing systems may begin to map thetarget object at (853). In some embodiments, such mapping may includereceiving passable world data of the target object. In some embodiments,mapping may include depth sensor analysis of the contours of the targetobject. In some embodiments, mapping may include building a mesh modelof the items within the field of view and referencing them for potentiallabeling. In some embodiments, the target object is not a specificobject within the field of view that may be captured by a depth sensor,but rather is a depth plane within the field of view itself.

In some embodiments, at (855) a controller analyzes the emitted lightcompared to the received light, such as under the Beer-Lambert law orthe optical density relationship (described below) or scatter pattern ofa calibration curve. In some embodiments, at (856) the compared lightproperties are referenced in an absorption database, either locallystored on the system or remotely accessed through the system, toidentify a characteristic of the target object such as the materialforming or a material property of the target object. In someembodiments, an absorption database may comprise saturation lightcharts, such as the one depicted in FIG. 4B, or may comprise calibrationcurves of particular light wavelengths.

In some embodiments, at (854) the radiation detectors do not receivelight of different wavelengths than the wavelength of the light emittedat (852), and a controller cannot conduct a spectroscopic analysis. Suchan occasion would occur as in FIG. 73, with overlap of wavelengths incertain ranges for oxygenated and deoxygenated blood. In someembodiments, at (854 a) no wavelength difference is detected between theemitted light and received light, and substep (854 b) initiates byemitting light at another different wavelength than that emitted at(852). The new light emitted and light received information is thendelivered to a controller at (855).

In some embodiments, real world cameras may additionally, subsequent tomapping a target object (853) and potentially concurrent with each of(852 through 856), identify subpixels within a field of field indicativeof irregularities at (857). For example, in some embodiments, colorcontrast between pixels is detected during real world capture at (853)and at (857) these pixels are further altered to highlight such contrastas potential unhealthy cells. In some embodiments, real world capture(853) detects irregular lines among pixel clusters and at (857) thepixels bounded by the irregular lines are marked (such as by a virtualcolor overlay) on a user display.

In some embodiments, method (850) terminates at (858) with the systemdisplaying the tissue or material property of the tissue to the user. Insome embodiments, display may comprise a textual label virtuallydisplayed proximate to the target object, an audio label describing thetarget object as determined from the absorption database (630), or avirtual image of similar tissue or object identified by absorptiondatabase (630) juxtaposed proximate to the target object.

In some embodiments, a significant amount of the spectroscopy activityis implemented with software operated by the controller (844), such thatan initial task of locating desired targets (e.g., blood vessels, muscletissue, bone tissue, or other tissue and at a desired depth) isconducted using digital image processing (such as by color, grayscale,and/or intensity thresholding analysis using various filters. Suchtargeting may be conducted using pattern, shape recognition or texturerecognition. Cancerous cells or otherwise irregular cells commonly haveirregular borders. A camera system may identify a series of pixelswithin a camera field of view (such as cameras 124 and field of view 18,22 of FIG. 5) with an irregular, non-linear pattern and prompt attentionto identify such as a border to a potentially unhealthy cell.Alternatively, the software and controller may be configured to use theintensity of the center of the targeted object and the intensity of thesurrounding objects/tissue to determine contrast/optical density withthe targeted object to determine abnormalities. Such measures may merelybe used to identify areas of interest for spectroscopic scan consistentwith this disclosure, and not necessarily a means of identifying tissueitself. Further, as previously described with reference to irregularcells (626) in FIG. 6, an augmented reality system may overlay a labelor color pattern within the borders of the potentially unhealthy cellsto flag them/highlight them against surrounding healthy cells.

In some embodiments, the controller (844) may be utilized to calculatedensity ratios (contrast) and to calculate the oxygen saturation fromthe density ratios of various pulse oximetry properties in bloodvessels. Vessel optical density (“O.D.”) at each of the two or moreemitted wavelengths may be calculated using the formula:ODvessel=−log₁₀(Iv/It)

wherein ODvessel is the optical density of the vessel; Iv is the vesselintensity; and It is the surrounding tissue intensity.

Oxygen saturation (also termed “SO2”) in a blood vessel may becalculated as a linear ratio of vessel optical densities (OD ratio, or“ODR”) at the two wavelengths, such that:SO₂=ODR=OD_(firstwavelength)/OD_(secondwavelength)

In one embodiment, wavelengths of about 570 nm (sensitive todeoxygenated hemoglobin) and about 600 nm (sensitive to oxygenatedhemoglobin) may be utilized in vessel oximetry, such that S02=ODR=OD₆₀₀nm/D570 nm; such formula does not account for adjusting the ratio by acalibration coefficient.

The above formulas are merely examples of references for calculatingmaterial properties. One of skill in the art will appreciate many othertissue properties and relationships a controller may determine.

It will be appreciated that utilizing the controller (844) to performcalculations and/or make determinations may involve performingcalculations locally on a processor within the controller (844). In someother embodiments, performing calculations and/or making determinationswith the controller (844) may involve utilizing the controller tointerface with external computing resources, e.g., resources in thecloud (46) such as servers (110).

Ambient Light Correction

As described above, the presence of ambient light in the environment maycomplicate the spectroscopic methods described herein. FIGS. 9A-9Cillustrate an effect ambient light may have on certain spectroscopicmethods. Each of the graphs of FIGS. 9A-9C depicts light intensity overtime as the spectroscopic methods disclosed herein are performed.

With reference to FIG. 9A, in some spectroscopy systems, a light sourcemay emit light (905) at a constant or substantially constant intensity.For example, the light (905) of constant intensity may be emitted by anyof the inward or outward-facing light sources described herein, such asemitters (126, 832, 834, FIG. 5) or light source (612, FIG. 6). Inembodiments described herein in which light is emitted at a plurality ofwavelengths, the intensity may correspond to one or more of theplurality of wavelengths.

The graph of FIG. 9B depicts example time-domain functions ofconstituent components of the light detected at an electromagneticradiation detector corresponding to the light emissions of FIG. 9A. Forexample, the light may be detected at any of the inward oroutward-facing electromagnetic radiation detectors described herein,such as detectors (126, 828, 830, FIG. 5) or photodetectors (614, FIG.6). As shown in FIG. 9B, the detected light includes a reflected lightcomponent (910) and an ambient light component (915). The reflectedlight component (910) includes a portion of the emitted light (905) thathas been reflected from the surface of a target object, such as theobject (620, FIG. 6). Generally, the reflected light component (910) hasa lower intensity relative to the emitted light (905), as some of theemitted light (905) may be absorbed or scattered in a differentdirection by the target object, or may pass by the target object.

The ambient light component (915) may have a constant or variableintensity, and may comprise light emitted from a source other than thelight source (126, 832, 834, FIG. 5; 612, FIG. 6). For example, theambient light component (915) may include light from the environmentthat has passed by the target object to be incident upon the detector,as well as light from the environment that has been scattered by thesurface of the target object. Depending on the particular environment inwhich the spectroscopic analysis is being performed and/or the materialcomposition of the target object (620), the intensity of the ambientlight component (915) may be greater than, equal to, or less than theintensity of the reflected light component (910), and/or may vary overtime. Without being bound by theory, the ambient light component (915)may introduce noise into the detection of the reflected light component(910) or may otherwise alter the measurement of that light. Withreference to FIG. 9C, this added noise may result in a singlemeasurement of detected light (920), in which it may be difficult toseparately identify the reflected light component (910) and the ambientlight component (915). For example, both the reflected light component(910) and the ambient light component (915) may include light havingsimilar wavelengths. Thus, with light emission and measurementcorresponding to FIGS. 9A-9C, it may be difficult to accuratelydetermine the amount of reflectance or absorbance of the emitted light(905) and associated material properties based on reflectance orabsorbance, since ambient light may skew the measurement of thereflected light.

Referring again to FIGS. 5 and 6, an example method of removing ambientlight noise using an anti-scatter grid will now be described. In someembodiments, noise caused by ambient light may be reduced and/or removedby including an anti-scatter grid (129, 629) in the head-mountedcomponent (58, FIG. 5) or spectroscopic array (126, FIG. 6). In someembodiments, ambient light may be understood to be similar to scatteredlight in the sense that the ambient light may propagate to a radiationdetector at angles different from reflected light. Consequently,preventing scattered light from reaching the radiation detector may beunderstood to reduce the contribution of ambient light to measurementsby the radiation detector.

With reference to FIGS. 5 and 6, the anti-scatter grid may be locatedanywhere within the head-mounted component (58, FIG. 5) or spectroscopicarray (126, FIG. 6) such that the anti-scatter grid (129, 629) liesalong the path of the reflected and ambient light between the targetobject and the electromagnetic radiation detectors (126, 828, 830, FIG.5; 614, FIG. 6). In some embodiments, the anti-scatter grid mayattenuate incident light that is not parallel to, or within a thresholdangle of, the direction of the emitted light (613) and the reflectedlight (615). In some embodiments, such threshold angle is less thanthirty-two degrees, though greater or smaller threshold angles arepossible given the desired level of filtering desired. For example, thethreshold angle may be less than sixteen degrees, less than forty-eightdegrees, less than sixty-four degrees, etc. One of skill in the art willrecognize the thirty-two degree threshold as a function of the sixteendegrees corresponding to a bundle of light rays consistent with outputlight emitting sources. Accordingly, the anti-scatter grid generallyallows the emitted and reflected light to pass therethrough with littleto no attenuation; much of the scattered light and ambient light may betraveling at relatively larger angles relative to the emitted light(613) and the reflected light (615), and may thus be attenuated at theanti-scatter grid (129, 629). For example, the surfaces of theanti-scatter grid exposed to ambient or reflected light may be coatedwith or formed of a material that has high levels of light absorption,particularly at the wavelengths relevant for a spectroscopic analysis.Examples of light absorbing materials include VANTABLACK®. Accordingly,the ambient light component of the detected light, as depicted in FIG.9B, may be substantially reduced, resulting in an absorbance orreflectance measurement that more accurately corresponds to theabsorbance or reflectance of the target object (620).

Referring now to FIGS. 10A-10C, an example method of removing ambientlight noise based on aspects of the emitted light will now be described.In various embodiments, the emitted light for spectroscopic analysis maybe conditioned such that the reflected light detected at theelectromagnetic radiation detectors (126, 828, 830, FIG. 5; 614, FIG. 6)may be distinguished from ambient or scattered light. In one example, asshown in FIGS. 10A-10C, the emitted light may be emitted intime-separated pulses (1005). In some embodiments, the pulses (1005) maybe separated by a period of no emission, or may be separated by a periodof emission at lower intensity relative to the pulses (1005). In someembodiments, the pulses (1005) may be produced by a component such as anoptical time-of-flight depth sensor or other component configured toproduce pulses of light.

With reference to FIG. 10B, at least a portion of the light emitted inpulses 1005 may be reflected by the target object (620) toward theelectromagnetic radiation detectors (126, 828, 830, FIG. 5; 614, FIG. 6)as reflected pulses (1010). It will be appreciated that the shape of thereflected pulses (1010) may be similar to the shape of the emittedpulses (1005). The reflected pulses (1010) may generally be of a lowerintensity relative to the emitted pulses 1005, as some of the emittedlight may be absorbed at the target object (620) and/or may pass by thetarget object (620).

Similar to the situation depicted in FIG. 9B, an ambient light component(1015) may have a constant or variable intensity, and may comprise lightemitted from a source other than the light source (126, 832, 834, FIG.5; 612, FIG. 6). For example, the ambient light component (1015) mayinclude light from the environment that has passed by the target objectto be incident upon the detector, as well as light from the environmentthat has been scattered by the surface of the target object. Dependingon the particular environment in which the spectroscopic analysis isbeing performed and/or the material composition of the target object(620), the intensity of the ambient light component (915) may be greaterthan, equal to, or less than the maximum intensity of the reflectedlight component (910), and/or may vary overtime. However, the ambientlight component (1015) generally will not have a regular pulse patternsimilar to the emitted pulses (1005) and reflected pulses (1010).

Thus, as shown in FIG. 10C, the final measurement of detected light(1020) retains a distinguishable ambient light component (1025) andpulses (1030). For example, where the ambient light component (1025) isroughly steady-state, measurements of light levels between the pulsesprovides a measurement of the intensity of the ambient light component(1025). Accordingly, an intensity and/or time-domain intensity functionof the ambient light component (1025) may be calculated. The ambientlight component (1025) may then be subtracted from the intensity of thepulses (1030) in the final measurement of detected light (1020) todetermine an ambient light-adjusted measurement of reflectance.

In some embodiments, the calculation of the ambient light intensity andadjustment of the intensity of pulses (1030) may be performed at one ormore components such as the controller (844) or processor (611) depictedand described elsewhere herein, and/or at any other processingcomponents in communication with the system. Such calculations and/ordeterminations may be performed locally and/or may involve utilizinglocal components to interface with external computing resources, e.g.,resources in the cloud (46) such as servers (110).

The previously described example of time-domain multiplexing usingregular pulses of emitted light is just one example of various methodsby which the emitted light may be made more readily distinguishable fromthe ambient light, relative to a constant output of light. Various othertypes of conditioned emitted light may equally be used without departingfrom the spirit or scope of the present disclosure. In variousembodiments, the emitted light may be varied in a known manner (e.g., aknown property of the light may have a known variance with time), andthe detected light may also be expected to vary in a similar manner.Consequently, light that does not vary in this expected manner may beunderstood to be light coming from another source, e.g., ambient light,and, as such, may be subtracted from the detected light.

In one example, the intensity of the emitted light may vary sinusoidallyat a constant frequency, such that a component of the detected light(920, 1020) varying at the same frequency may be identified and isolatedto determine an ambient light-adjusted reflectance (with the assumptionthat the ambient light levels are roughly constant, while the varianceof the emitted light intensity with time is known). In another example,frequency and/or amplitude of the emitted light signal may be modulatedin a known way, e.g., frequency and/or amplitude may vary with a knowndependence upon time. A frequency associated with the emitted light mayfurther be produced by heterodyning. In yet another example, any ofvarious coding schemes may be incorporated into the emitted light, suchas a Manchester code, a Hamming code, or the like. In another example,any of various pseudo-random variations may be incorporated into theemitted light signal. The detected light may be correlated with thepseudo-random variations to identify reflected and ambient components ofthe detected light (920, 1020). In yet another example, the emittedlight may be polarized (e.g., using a linear, circular, or ellipticalpolarization) in a polarization state not expected to be present orprevalent in the ambient and/or scattered light, such that the reflectedand ambient components of the measured light may be isolated based onpolarization state.

Referring jointly to FIGS. 5, 6, and 9A-10, another example method ofcorrecting for ambient light noise will now be described. In someembodiments, the head-mounted component (58, FIG. 5) or spectroscopicarray (126, FIG. 6) includes an ambient light detector (128, FIG. 5;628, FIG. 6). The ambient light detector (128, 628) may include any oneor more of the various types of electromagnetic radiation detectorsdescribed here. The ambient light detector (128, 628) may beinward-facing and/or outward-facing, for example, in the direction asemitted light for the spectroscopic analysis. In some embodiments, theambient light detector (128, 628) may be oriented in the same or similardirection as the electromagnetic radiation detector(s) (126, 828, 830)so as to capture the ambient light in the same general direction as thetarget object. In similar environments, the ambient light detector (128,628) a point in a direction different from the target object, therebyproviding a measurement of ambient light levels that are independent oflight reflected from the target object.

It will be appreciated that the various ambient light leveldeterminations made above provide an ambient light correction which maybe applied to measurements of detected reflected light to remove thecontribution of ambient light to the detected reflected lightmeasurement. For example, the intensity values of the ambient light maysimply be subtracted from the intensity values of the light detected bya radiation detector to arrive at the corrected reflected lightmeasurement. In addition, as described herein, the difference betweenthe intensities of the emitted light and the detected reflected lightprovide an absorbance measurement for the target object, which may beutilized to determine properties of the object as described herein. Anyof the ambient light correction methods described above may be usedindividually and/or together in any combination. For example, in someembodiments the time-encoding or ambient light detection methodsdescribed above may be implemented in systems that additionally includean anti-scatter grid.

FIG. 11 illustrates a method (1100) for using a wearable AR/VR systemfeaturing spectroscopy components to identify a material or propertieswithin a material, including the ambient light correction methodsdescribed herein. In some embodiments, the method may optionally beginat (1102) with the system detecting ambient light in the vicinity of atarget object to be imaged. As described above, the system may detectthe ambient light at an ambient light sensor and/or any electromagneticradiation detectors of the system. For example, the system may take 1,2, 5, 10, or more discrete measurements of the ambient light, and/or maycontinuously measure the ambient light while the light sources of thesystem are not emitting an irradiation pattern. Measurements of theambient light provides an ambient light correction, which may be laterapplied to correct measurements of reflected light. After the optionalambient light detection at (1102), the method (1100) continues asdescribed above with reference to FIG. 8.

The various time-domain encoding methods described above with referenceto FIGS. 10A-10C may be incorporated at (1110), in which the controllerdetermines one or more properties of light from the emitted light andthe reflected light data. For example, at (1110), the system may detectlight travelling from the direction of the target object and compare thedetected light to the conditioned light emitted at (1106) to identifyany pattern, pulse, encoding, polarization, or other identifyingcharacteristic of the emitted light, as described herein. In someembodiments, at (1110), the display system may also apply the ambientlight correction to adjust measurements of reflected light obtained at(1108) before making a final determination of the light property.

Accordingly, with the enhanced ambient light correction methodsdescribed herein, the system at (1108 a) may be able to more effectivelydetermine whether there is a difference between the emitted light andthe reflected light. Similarly, the system at (1110) may be able to moreaccurately determine an absorption or reflection characteristic of thetarget object, thus enhancing the accuracy of the reference to theabsorption database at (1112) and determination of material property fordisplay at (1116).

Computer Vision

As discussed above, the spectroscopy system may be configured to detectobjects in or features (e.g. properties) of objects in the environmentsurrounding the user. In some embodiments, objects or properties ofobjects present in the environment may be detected using computer visiontechniques. For example, as disclosed herein, the spectroscopy system'sforward-facing camera may be configured to image an object and thesystem may be configured to perform image analysis on the images todetermine the presence of features on the objects. The system mayanalyze the images, absorption determinations, and/or reflected and/orscattered light measurements acquired by the outward-facing imagingsystem to object recognition, object pose estimation, learning,indexing, motion estimation, or image restoration, etc. One or morecomputer vision algorithms may be selected as appropriate and used toperform these tasks. Non-limiting examples of computer vision algorithmsinclude: Scale-invariant feature transform (SIFT), speeded up robustfeatures (SURF), oriented FAST and rotated BRIEF (ORB), binary robustinvariant scalable keypoints (BRISK), fast retina keypoint (FREAK),Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm,Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneouslocation and mapping (vSLAM) techniques, a sequential Bayesian estimator(e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

As discussed herein, the objects or features (including properties) ofobjects may be detected based on one or more criteria (e.g., absorbance,light reflection, and/or light scattering at one or more wavelengths).When the spectroscopy system detects the presence or absence of thecriteria in the ambient environment using a computer vision algorithm orusing data received from one or more sensor assemblies (which may or maynot be part of the spectroscopy system), the spectroscopy system maythen signal the presence of the object or feature.

One or more of these computer vision techniques may also be usedtogether with data acquired from other environmental sensors (such as,e.g., microphone, GPS sensor) to detect and determine various propertiesof the objects detected by the sensors.

Machine Learning

A variety of machine learning algorithms may be used to learn toidentify the presence of objects or features of objects. Once trained,the machine learning algorithms may be stored by the spectroscopysystem. Some examples of machine learning algorithms may includesupervised or non-supervised machine learning algorithms, includingregression algorithms (such as, for example, Ordinary Least SquaresRegression), instance-based algorithms (such as, for example, LearningVector Quantization), decision tree algorithms (such as, for example,classification and regression trees), Bayesian algorithms (such as, forexample, Naive Bayes), clustering algorithms (such as, for example,k-means clustering), association rule learning algorithms (such as, forexample, a-priori algorithms), artificial neural network algorithms(such as, for example, Perceptron), deep learning algorithms (such as,for example, Deep Boltzmann Machine, or deep neural network),dimensionality reduction algorithms (such as, for example, PrincipalComponent Analysis), ensemble algorithms (such as, for example, StackedGeneralization), and/or other machine learning algorithms. In someembodiments, individual models may be customized for individual datasets. For example, the wearable device may generate or store a basemodel. The base model may be used as a starting point to generateadditional models specific to a data type (e.g., a particular user), adata set (e.g., a set of absorbance, light reflection, and/or lightscattering values obtained at one or more wavelengths), conditionalsituations, or other variations. In some embodiments, the spectroscopysystem may be configured to utilize a plurality of techniques togenerate models for analysis of the aggregated data. Other techniquesmay include using pre-defined thresholds or data values.

The criteria for detecting an object or feature of an object may includeone or more threshold conditions. If the analysis of the data acquiredby a sensor (e.g., a camera or photodetector) indicates that a thresholdcondition is passed, the spectroscopy system may provide a signalindicating the detection the presence of the object in the ambientenvironment. The threshold condition may involve a quantitative and/orqualitative measure. For example, the threshold condition may include ascore or a percentage associated with the likelihood of the objectand/or feature being present. The spectroscopy system may compare thescore calculated from the sensor's data with the threshold score. If thescore is higher than the threshold level, the spectroscopy system maysignal detection of the presence of an object or object feature. In someother embodiments, the spectroscopy system may signal the absence of theobject or feature if the score is lower than the threshold.

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. A code module maybe compiled and linked into an executable program, installed in adynamic link library, or may be written in an interpreted programminglanguage. In some embodiments, particular operations and methods may beperformed by circuitry that is specific to a given function. In someembodiments, the code modules may be executed by hardware in thecontroller (844) (FIG. 5) and/or in the cloud (46) (e.g., servers(110)).

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (70, FIG. 2C), the remoteprocessing module (72, FIG. 2D), and remote data repository (74, FIG.2D). The methods and modules (or data) may also be transmitted asgenerated data signals (e.g., as part of a carrier wave or other analogor digital propagated signal) on a variety of computer-readabletransmission mediums, including wireless-based and wired/cable-basedmediums, and may take a variety of forms (e.g., as part of a single ormultiplexed analog signal, or as multiple discrete digital packets orframes). The results of the disclosed processes or process steps may bestored, persistently or otherwise, in any type of non-transitory,tangible computer storage or may be communicated via a computer-readabletransmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless specifically stated otherwise. Inother words, use of the articles allow for “at least one” of the subjectitem in the description above as well as claims associated with thisdisclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element-irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

What is claimed is:
 1. A wearable spectroscopy system comprising: ahead-mounted display system removably mountable on a user's head; one ormore head-mounted light sources coupled to the head-mounted displaysystem and configured to emit light in an irradiated field of view; oneor more head-mounted electromagnetic radiation detectors coupled to thehead-mounted display system and configured to receive reflected lightfrom a target object irradiated by the one or more light sources withinthe irradiated field of view; one or more processors; and one or morecomputer storage media storing instructions that, when executed by theone or more processors, cause the system to perform operationscomprising: causing the one or more light sources to emit light; causingthe one or more electromagnetic radiation detectors to detect light fromthe irradiated field of view including the target object; determining anambient light correction by detecting ambient light levels; applying theambient light correction to the detected light to determine levels oflight absorption related to the emitted light and reflected light fromthe target object; identifying, based on the levels of light absorption,a characteristic of the target object; and displaying an outputassociated with the characteristic of the target object to the user onthe head-mounted display system.
 2. The system of claim 1, furthercomprising an absorption database of light absorption properties of aplurality of materials.
 3. The system of claim 1, wherein the ambientlight correction comprises one or more of: an ambient light intensityvalue, an average of a plurality of ambient light intensity values, amedian of a plurality of ambient light intensity values, and atime-domain ambient light intensity function.
 4. The system of claim 1,further comprising at least one eye tracking camera configured to detecta gaze of the user, wherein the irradiated field of view issubstantially in the same direction as the detected gaze.
 5. The systemof claim 1, wherein the one or more electromagnetic radiation detectorsare further configured to detect the ambient light levels.
 6. The systemof claim 1, further comprising an ambient light detector coupled to thehead-mounted display system and configured to capture ambient light notemitted by the one or more light sources, the ambient light includingone or more wavelengths emitted by the one or more light sources.
 7. Thesystem of claim 6, wherein the ambient light detector comprises at leastone of a photodiode, a photodetector, and a digital camera sensor. 8.The system of claim 6, wherein the instructions, when executed by theone or more processors, further cause system to perform operationscomprising: causing the ambient light detector to detect light while theone or more light sources are not emitting light; and determining theambient light correction based at least in part on the light detected bythe ambient light detector.
 9. The system of claim 1, further comprisingan anti-scatter grid coupled to the head-mounted display system betweenthe target object and the one or more electromagnetic radiationdetectors, the anti-scatter grid aligned to attenuate at least a portionof scattered light and ambient light incident upon the anti-scattergrid.
 10. The system of claim 9, wherein the anti-scatter grid isfurther disposed between the target object and a detector for detectingambient light levels.
 11. The system of claim 1, wherein the one or morelight sources are configured to emit the light in a series oftime-separated pulses, and wherein the instructions, when executed bythe one or more processors, further cause the system to performoperations comprising: identifying time-separated pulses of reflectedlight corresponding to the time-separated pulses emitted by the one ormore light sources; and determining the ambient light correction basedat least in part on an intensity of light detected at the one or moreelectromagnetic radiation detectors between the time-separated pulses ofreflected light.
 12. The system of claim 11, wherein the time-separatedpulses of reflected light are detected at the one or moreelectromagnetic radiation detectors.
 13. The system of claim 1, whereinthe one or more electromagnetic radiation detectors comprises at leastone of a photodiode and a photodetector.
 14. The system of claim 1,wherein the one or more electromagnetic radiation detectors comprises adigital image sensor.
 15. The system of claim 1, further comprising aninertial measurement unit positional system.
 16. The system of claim 15,wherein the inertial measurement unit positional system determines apose orientation of the user's head.
 17. The system of claim 16, whereinthe irradiated field of view is at least as wide as the poseorientation.
 18. The system of claim 1, wherein the head-mounted displaysystem comprises a waveguide stack configured to output light withselectively variable levels of wavefront divergence.
 19. A wearablespectroscopy system comprising: a head-mounted display removablymountable on a user's head; one or more head-mounted light sourcescoupled to the head-mounted display system and configured to emit lightin an irradiated field of view; one or more head-mounted electromagneticradiation detectors coupled to the head-mounted display system andconfigured to receive reflected light from a target object irradiated bythe one or more light sources within the irradiated field of view; oneor more processors; and one or more computer storage media storinginstructions that, when executed by the one or more processors, causethe system to perform operations comprising: detecting ambient light ofa first wavelength within the irradiated field of view; emitting lightof the first wavelength toward the target object; detecting light of thefirst wavelength reflected by the target object; subtracting anintensity of the detected ambient light of the first wavelength from anintensity of the detected light reflected by the target object tocalculate a level of light absorption related to the emitted light andthe reflected light from the target object; identifying, based on anabsorption database of light absorption properties of a plurality ofmaterials, a material characteristic of the target object; anddisplaying, to the user, an output associated with the materialcharacteristic.
 20. A wearable spectroscopy system comprising: ahead-mounted display removably mountable on a user's head; one or morehead-mounted light sources coupled to the head-mounted display systemand configured to emit light in an irradiated field of view; one or morehead-mounted electromagnetic radiation detectors coupled to thehead-mounted display system and configured to receive reflected lightfrom a target object irradiated by the one or more light sources withinthe irradiated field of view; one or more processors; and one or morecomputer storage media storing instructions that, when executed by theone or more processors, cause the system to perform operationscomprising: emitting light of a first wavelength in an irradiated fieldof view, the light comprising a time-encoded variation; detecting lightof the first wavelength reflected from a target object within theirradiated field of view; identifying, based at least in part on thedetected light and the time-encoded variation, an ambient lightcomponent of the detected light and a reflected component of thedetected light; identifying, based at least in part on the reflectedcomponent and an absorption database of light absorption properties ofat least one material, a material characteristic of the target object;and displaying, to the user, an output associated with the materialcharacteristic.